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FOREWORD 


As  the  Army’s  lead  laboratory  for  research,  development,  and  analysis  on  training  and  other  human 
dimensions  of  operational  performance,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  is  poised  to  address  emerging  topics  as  requested  by  Army  organizations.  One 
such  emerging  topic  is  lifelong  learning,  a  concept  adopted  by  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC)  to  fundamentally  change  the  timing  and  accessibility  of  Army  schoolhouse 
training  and  education.  ARI  has  supported  the  development  of  the  lifelong  learning  concept  by 
funding  an  initial  investigation  into  the  metrics  necessary  for  assessing  the  effectiveness  and  impact 
of  lifelong  learning  centers  (LLCs),  the  physical  instantiation  of  the  lifelong  learning  concept. 

The  present  research  was  conducted  at  the  request  of  TRADOC’s  Training  Development  and 
Delivery  Directorate  to  follow  up  the  previous  ARI-funded  exploration  of  LLC  assessment.  The  goal 
of  the  present  effort  was  threefold:  (1)  to  ensure  that  the  LLC  Assessment  Framework  initially 
developed  would  be  broadly  applicable  across  current  and  future  LLCs  that  provide  military 
operational  specialty  qualification  education;  (2)  to  conduct  an  assessment  of  the  LLC  implemented 
at  Fort  Gordon,  GA;  and  (3)  to  draw  up  a  plan  for  continued  LLC  self-assessment. 

This  final  report  presents  in  detail  the  research  conducted  and  its  findings.  A  revised  LLC 
Assessment  Framework  is  provided  with  the  changes  and  their  associated  justifications 
documented.  A  narrative  of  the  Fort  Gordon  LLC  assessment  method  and  findings  is  presented, 
as  are  recommendations  strengthening  the  LLC  in  order  to  achieve  optimal  impact.  Finally,  a 
proposed  plan  for  LLC  self-assessment  is  provided  along  with  a  rationale  for  its  design.  Part  of 
this  work  (the  Fort  Gordon  LLC  assessment  findings)  was  presented  to  the  Fort  Gordon 
Directorate  of  Training  in  December  2007.  The  complete  body  of  work  will  be  presented  to  the 
next  Council  of  Colonels  to  be  held  to  discuss  TRADOC’s  lifelong  learning  initiative. 


MICHELLE  SAMS,  Ph.D. 
Director 
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EXECUTIVE  SUMMARY 


Research  Requirement: 

The  lifelong  learning  concept  has  been  advanced  as  an  Army-wide  solution  to  the 
problem  of  meeting  the  educational  demands  of  a  rapidly  changing  operational  environment. 
Lifelong  learning  is  defined  as  “a  mixture  of  traditional  schoolhouse  resident  education  with 
education  presented  in  other  locations  at  the  individual’s  teachable  moment”  (TRADOC,  2004). 
Lifelong  Learning  Centers  (LLCs)  comprise  a  suite  of  technologies  that  enable,  among  other 
things,  online  posting  of  schoolhouse  curriculum  materials,  courseware  downloads,  and 
distributed  collaboration  among  users. 

Previous  research  by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  established  a  process  for  assessing  the  effectiveness  of  LLCs  in  enhancing 
learning  and  readiness.  The  final  report  for  that  effort  presented  a  comprehensive  framework  for 
conceptualizing  how  resources  invested  in  LLCs  could  produce  change  to  instruction,  learning, 
and  organizational  effectiveness.  The  framework  was  used  to  conduct  a  formative  assessment  of 
a  pilot  LLC  established  at  Fort  Leavenworth  to  deliver  leader  education,  which  demonstrated  that 
the  framework  was  a  feasible  and  useful  tool  for  conducting  LLC  program  evaluation. 

The  limited  scope  of  the  previous  ARI  effort,  however,  prevented  an  explicit  test  of  the 
generalizability  of  the  assessment  framework.  It  remains  to  be  determined  whether  the 
framework  applies  as  expected  to  other  LLCs,  especially  those  that  deliver  military  operational 
specialty  (MOS)  qualification  instruction.  New  metrics  and  measures  associated  with  the  range 
of  training  and  education  strategies  provided  by  LLCs,  if  required,  must  be  developed.  The 
present  research  sought  to  examine  the  generalizability  of  the  LLC  Assessment  Framework  and 
to  apply  the  framework  to  conducting  a  formative  assessment  of  the  Fort  Gordon  LLC.  This 
effort  also  investigated  the  requirements  for  enabling  LLCs  to  conduct  self-assessment  such  that 
the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  could  leverage  ARI  research  to 
further  the  development  of  its  lifelong  learning  initiative. 

Procedure: 

An  in-depth  set  of  interviews  and  focus  groups  were  conducted  with  the  large  variety  of 
stakeholders  involved  with  the  Fort  Gordon  LLC,  a  well-established  MOS-based  LLC  and  the 
prototype  for  the  lifelong  learning  concept.  The  training  and  education  strategies  enabled  by  the 
Fort  Gordon  LLC  represent  the  majority  of  learning  strategies  supported  by  other  current  and 
anticipated  LLCs.  The  information  gathered  was  used  to  determine  the  applicability  of  the 
assessment  framework  presented  in  Cianciolo  (2007)  and  to  make  modifications  to  enhance 
generalizability  where  necessary.  The  revised  framework  was  used  to  assess  the  learning 
effectiveness  and  readiness  impact  of  the  Fort  Gordon  LLC.  Six  aspects  of  the  Fort  Gordon  LLC 
were  examined:  MOSQ  Instruction,  Assignment-Oriented  Training,  Simulations,  Discussion 
Forums,  Leader  Education,  and  On-Demand  Training.  Assessment  involved  interviews,  focus 
groups,  classroom  observations,  system  analysis,  and  archival  data  review  where  access  to 
resources  (e.g.,  operational  units)  for  collecting  data  was  limited.  Requirements  and  methods  for 
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conducting  LLC  self-assessment  were  determined  via  a  combination  of  interviews,  literature 
review,  and  examination  of  LLC  assessment  lessons  learned. 

Findings: 

The  LLC  Assessment  Framework  proved  to  be  largely  generalizable  across  different 
types  of  LLCs,  but  some  modification  was  necessary  in  order  to  reflect  additional  determinants 
of  organizational  impact.  These  additional  determinants  were  factors  external  to  the  sphere  of 
influence  of  the  LLC,  chiefly  personnel  management  and  institutional  training  procedures,  which 
can  moderate  the  relation  between  outputs  and  outcomes.  Some  modification  also  was  made  to 
metrics  and  measures  in  order  to  include  more  generalizable  and  usable  methods  for  capturing 
LLC  outcomes. 

The  assessment  of  the  Fort  Gordon  LLC  revealed  that,  overall,  the  outputs  necessary  to 
achieve  educational  transformation  and  impact  (e.g.,  downloadable  computer-based  training 
products)  were  produced  by  the  LLC  staff  and  affiliated  stakeholders.  Where  expected  outputs 
were  not  observed,  closer  coordination  between  the  LLC  and  stakeholders  in  the  proponent 
schoolhouse  appeared  to  be  the  necessary  remedy.  The  link  between  outputs  and  outcomes  was 
not  readily  observed  in  the  present  research  effort.  Larger  organizational  factors,  such  as  Army 
policy  regarding  who  may  train  which  MOSs,  played  a  role  in  limiting  impact  by  preventing 
uniform  access  to  instructional  materials,  reducing  the  time  available  to  conduct  training 
anytime,  anywhere,  and  constraining  the  adaptivity  of  training  systems  to  evolve  in  response  to 
changing  educational  requirements. 

Analysis  of  LLC  self-assessment  requirements  indicated  that  assessment  could  not  be 
readily  automated.  Rather,  the  evaluation  process  requires  people  knowledgeable  of  educational 
theory  and  technology  and  capable  of  having  extensive  face-to-face  contact  with  LLC 
stakeholders  within  and  outside  of  the  schoolhouse.  This  requirement  may  be  met  by  relatively 
minor  modifications  to  current  staffing  levels  at  the  program  management  level.  Moreover, 
existing  institutional  resources  may  be  leveraged  to  support  the  assessment  process. 

Utilization  and  Dissemination  of  Findings: 

Assessing  LLCs  informs  decision  making  by  shedding  light  on  the  educational  impact  of 
financial  investment  in  lifelong  learning.  The  revised  framework  for  LLC  assessment  reveals  the 
factors  outside  of  the  immediate  LLC  context  that  can  influence  the  link  between  the  outputs  of 
the  initiative  and  the  outcomes  it  achieves.  In-depth  assessment  is  necessary  to  demonstrate  how 
existing  investment  and  processes  may  be  better  leveraged  to  maximize  the  benefit  of  the  dollars 
spent.  Without  assessment,  decisions  to  save  cost  may  needlessly  reduce  the  effectiveness  of 
education,  which  may  in  turn  affect  readiness  and  morale.  The  present  research  provides 
recommendations  for  leveraging  the  capabilities  inherent  in  LLCs  and  for  addressing  external 
influences  to  success  such  that  future  staffing  and  development  decisions  enhance  the  impact  of 
the  lifelong  learning  initiative. 
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INTRODUCTION 


“Like  many  units,  I  suspect  we  received  a  lot  of  equipment  that  no  one  in  the  company  had  ever 
seen  or  used:  Polish  Mine  Boots  ...  Talon  Robots,  civilian  global  positioning  systems  and  laser 
range  finders.  Falcon  View  Mapping  Software  ...  frequency  jammers,  and  Polaris  and  John  Deer 
all-terrain  vehicles,  to  give  a  few  examples.”  -  MSG  C.  Peterson 

The  Soldiers  in  today’s  Army  face  unprecedented  levels  of  complexity  in  their  operating 
environment.  The  particular  challenges  of  asymmetric  warfare  and  counterinsurgency  are  well 
recognized  and  documented  (e.g.,  FM  3-24:  Counterinsurgency,  see  also  Chiarelli  &  Michaelis, 
2005;  Williams,  2003),  but  as  the  above  quotes  from  Long  Hard  Road:  NCO  Experiences  in 
Afghanistan  and  Iraq  (2007)  illustrate,  these  challenges  only  partially  represent  the  body  of 
knowledge  and  skills  that  must  be  acquired  to  achieve  success.  The  Army  itself  is  rapidly 
changing,  adopting  new  command,  control,  and  communications  technologies,  new  unit 
structures  and  personnel  management  practices,  and  new  performance  objectives.  Moreover,  the 
politically  sensitive  and  highly  public  nature  of  actions  taken  by  U.S.  Soldiers  significantly 
broadens  the  impact  of  mistakes.  Consequently,  Soldiers  must  develop  the  capability  to  execute 
more  difficult  tasks,  more  often,  and  under  greater  pressure. 

The  Army’s  institutional  training  and  education  system  has  changed  in  response  to  the 
demands  of  the  modem  operating  environment.  For  instance,  the  Army  Distance  Learning 
Program  (now  called  the  Army  Distributed  Learning  Program)  was  initiated  in  the  1 990s  to 
facilitate  the  participation  of  students  outside  the  schoolhouse  in  standardized  institutional 
education.  The  anticipated  benefits  of  distance  learning  included  an  increased  graduation  rate, 
reduced  course  duration,  enhanced  personnel  readiness  status,  and  cost  savings  (Leonard, 
Winkler,  Hove,  Ettedgui,  Shanley,  Sollinger,  2001;  Shanley,  Leonard,  &  Winkler,  2001). 

A  limitation  of  the  distance  learning  program,  however,  was  that  it  was  yoked  to  the 
institutional  timeframe  for  updating  and  providing  formal  education.  It  enhanced  the  formal 
education  process,  but  was  not  designed  to  address  the  learning  needs  of  Soldiers  outside  of  this 
context.  Students  continued  to  receive  instruction  in  lock-step  with  career  milestones  rather  than 
conducting  training  on  an  as-needed  basis  in  response  to  rapid  changes  in  the  operational 
environment.  A  new  approach  was  required  to  enable  training  and  education  that  both  met 
rigorous  standards  for  content  and  instructional  strategy  and  enabled  the  development  of  just-in- 
time  competency  throughout  a  Soldier’s  career. 

Overview  of  the  Lifelong  Learning  Initiative 

The  U.S.  Training  and  Doctrine  Command’s  (TRADOC’s)  lifelong  learning  concept  has 
emerged  as  this  approach.  Lifelong  learning  is  defined  as  “a  mixture  of  traditional  schoolhouse 
resident  education  with  education  presented  in  other  locations  at  the  individual’s  teachable 
moment”  (TRADOC,  2004).  Simply  stated,  the  purpose  of  the  lifelong  learning  concept  is  to 
fundamentally  change  the  way  Soldiers  interact  with  their  proponent  schoolhouse.  Its  intent  is  to 
enable  anytime,  anywhere  access  to  institutionally  approved  learning  content  by  leveraging 
information  technology  and  advanced  instructional  strategies,  including  performance  assessment 
and  after  action  reviews  (Wilson  &  Helms,  2003).  Such  access  not  only  would  benefit  Soldiers, 
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but  the  schoolhouse  as  well,  by  linking  the  operating  and  generating  forces  in  a  rapid  cycle  of 
learning  content  development  and  distribution. 

The  seed  for  lifelong  learning  was  planted  at  the  U.S.  Army  Signal  Center,  located  at  Fort 
Gordon,  GA,  which  sought  to  meet  the  training  requirements  of  communications  Soldiers  whose 
equipment  was  updated  far  more  frequently  than  formal  education  could  be  offered.  The  lifelong 
learning  concept  was  later  developed  as  an  Army-wide  solution  to  the  problem  of  meeting  the 
educational  demands  of  a  rapidly  changing  operational  environment.  The  long-term  vision  for 
the  lifelong  learning  initiative  is  to  enable  the  delivery  of  anytime,  anywhere  instruction  by  all 
TRADOC  schoolhouses. 

Lifelong  Learning  Centers  (LLCs)  are  the  concrete  instantiation  of  the  lifelong  learning 
concept,  the  portals  through  which  globally  distributed  learners  reach  back  to  the  institution. 

They  provide  web-based  access  to  the  instruction  provided  by  the  proponent  schoolhouse.  LLCs 
comprise  a  suite  of  technologies  that  enable,  among  other  things,  online  posting  of  schoolhouse 
curriculum  materials,  courseware  downloads,  and  distributed  collaboration  among  users.  In 
addition  to  the  Fort  Gordon  LLC,  several  LLCs  have  been  established  at  other  schoolhouses, 
including  the  Command  and  General  Staff  College  (Fort  Leavenworth)  and  the  Maneuver 
Support  Center  (Fort  Leonard  Wood),  among  others. 

Currently,  the  technical  management  of  LLCs  occurs  on  site  with  the  supported 
proponent.  Future  plans  include  consolidating  technical  management  into  one  Enterprise  LLC 
located  at  Fort  Eustis,  with  supporting  regional  hubs  on  location  with  the  original  three  LLCs 
(listed  above)  plus  a  few  additional  locations.  Such  integration  is  expected  to  reduce  costs  and 
facilitate  management  by  centralizing  technical  support,  content  management,  and  user 
development  (i.e.,  course  developers,  instructors,  discussion  facilitators,  etc.).  In  concert  with  the 
evolving  processes  of  technical  management  and  instructional  delivery,  assessment  strategies  for 
capturing  LLC  effectiveness  and  impact  also  have  developed  (Cianciolo,  2007). 

Background  of  the  Present  Research 

The  present  research  follows  from  a  previous  ARI  effort  to  explore  metrics  and  methods 
for  assessing  LLCs  (Cianciolo,  2007).  Initial  attempts  by  LLC  program  managers  and  staff  to 
develop  metrics  for  success  focused  on  the  activities  conducted  by  LLC  personnel  (e.g.,  number 
of  courses  placed  online)  and  other  easily  quantifiable  data  (e.g.,  number  of  students  enrolled). 
Although  such  metrics  were  important  reflections  of  personnel  task  execution,  they  did  not 
address  larger  questions  of  interest  to  decision  makers  at  the  organizational  level.  Unanswered 
questions  centered  on  the  “so  what?”  of  implementing  LLCs.  Did  LLCs  enhance  the  efficiency 
and  effectiveness  of  instruction?  Did  they  improve  readiness?  Did  they  reduce  the  costs  of 
education?  ARI  sought  to  develop  a  generalizable  framework  for  assessing  LLCs  that  could  be 
used  to  answer  these  questions.  A  secondary  purpose  of  that  investigation  was  to  conduct  a 
formative  assessment  of  the  pilot  LLC  established  at  the  Command  and  General  Staff  College, 
Fort  Leavenworth. 

As  reported  in  Cianciolo  (2007),  a  combination  of  literature  review,  interaction  with 
stakeholders,  and  data  collection,  was  used  to  perform  the  research.  Specifically,  a 
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comprehensive  literature  review  on  the  lifelong  learning  initiative  (e.g.,  strategic  plans, 
marketing  literature,  progress  reports,  etc.)  was  conducted,  as  was  a  review  of  the  scientific  and 
professional  literature  on  technology-assisted  instruction,  program  evaluation,  and  organizational 
behavior.  Interviews  and  focus  groups  were  conducted  with  a  variety  of  stakeholders  involved 
with  the  Fort  Leavenworth  LLC,  including  program  managers  and  leadership,  technical  staff, 
instructors,  course  developers,  and  students.  Where  possible,  input  from  people  involved  with  the 
broader  lifelong  learning  concept,  such  as  representatives  of  the  initiative’s  executive  agent,  also 
was  collected.  Following  the  literature  review  and  interviews,  surveys  were  administered  to 
provide  quantitative,  representative  data  on  the  implementation  and  effectiveness  of  the  Fort 
Leavenworth  LLC.  This  research  produced  a  comprehensive,  generalizable  framework  for 
conceptualizing  how  resources  invested  in  LLCs  can  produce  change  to  instruction,  learning,  and 
organizational  performance.  The  formative  assessment  of  the  Fort  Leavenworth  LLC 
demonstrated  that  the  framework  was  a  feasible  and  useful  tool  for  conducting  LLC  program 
evaluation.  Using  the  framework,  the  influence  of  the  LLC  technologies  on  instructional 
efficiency  and  effectiveness  relative  to  other  factors  could  be  identified. 

An  important  limitation  of  the  initial  LLC  assessment  research  was  that  the  design  of  the 
assessment  framework  was  based  on  an  unrepresentative  sample  of  LLCs  (N  =  1,  Fort 
Leavenworth).  The  small  size  of  this  sample  prevented  an  explicit  test  of  the  generalizability  of 
the  LLC  Assessment  Framework.  The  purpose  of  the  Fort  Leavenworth  LLC  was  to  support  the 
delivery  of  field  grade  leader  education,  but  other  LLCs  were  established  (or  have  been 
conceptualized)  to  enable  junior  leader  education,  military  operating  specialty  qualification 
(MOSQ)  instruction,  and/or  military  graduate-level  education  (e.g.,  judge  advocate  instruction). 

It  therefore  remains  to  be  determined  whether  the  framework  applies  as  expected  to  these  other 
LLCs,  particularly  those  that  deliver  MOSQ  instruction.  Leader  and  advanced  military  education 
vice  MOS-qualification  instruction  have  different  learner  audiences  and  require  different 
learning  environments,  instructional  strategies,  and  performance  assessment  and  feedback 
methods.  Moreover,  the  instruction  supported  by  MOS-based  LLCs  may  have  a  more  direct 
impact  on  mission  readiness  (e.g.,  through  the  just-in-time  development  or  retraining  of  skills 
necessary  to  perform  mission  essential  tasks)  and  potentially  broader  implications  for  cost 
savings  (e.g.,  by  using  simulations  as  a  substitute  for  expensive  hands-on  training  equipment). 

Overview  of  the  Present  Research 

The  present  research  was  sought  by  TRADOC  in  order  to  extend  the  findings  of  the 
previous  ARI  research  and  to  leverage  ARI  support  for  continued  development  of  the  lifelong 
learning  initiative.  Its  first  objective  was  to  examine  the  generalizability  of  the  LLC  Assessment 
Framework  and  to  revise  the  framework  to  ensure  its  broad  applicability.  To  meet  this  goal,  the 
sample  size  of  LLCs  considered  was  expanded  to  two—the  Fort  Leavenworth  LLC  and  the  Fort 
Gordon  LLC.  Although  still  small,  this  sample  size  represented  two-thirds  of  the  population  of 
well-established  LLCs  and  addressed  the  distinct  needs  of  MOS-based  LLCs  relative  to  LLCs 
focused  on  leader  education.  Selecting  the  Fort  Gordon  LLC  also  enabled  detailed  analysis  of  the 
broad  range  of  training  and  education  strategies  that  LLCs  could  offer,  including  simulation 
downloads  and  training  on  demand.  The  range  of  learning  strategies  featured  in  the  Fort  Gordon 
LLC  was  expected  to  represent  the  majority  of  strategies  applied  by  current  and  future  LLCs.  An 
in-depth  examination  of  the  Fort  Gordon  LLC  was  conducted  via  interviews,  literature  review 
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(where  applicable),  and  program  analysis.  The  ways  in  which  the  assessment  framework  had  to 
be  modified  to  address  the  diverse  needs  of  LLCs  were  identified  and  integrated  into  a  revised 
framework. 

The  second  objective  of  this  research  was  to  assess  the  effectiveness  and  impact  of  the 
Fort  Gordon  LLC.  Six  aspects  of  the  Fort  Gordon  LLC  were  examined:  MOSQ  Instruction, 
Assignment-Oriented  Training,  Simulations,  Discussion  Forums,  Leader  Education,  and  On- 
Demand  Training.  Data  were  collected  using  a  variety  of  methods,  including  interviews  and 
focus  groups,  classroom  observation,  archival  data  analysis,  and  system  analysis.  Data  collection 
alternatives  to  surveys  were  emphasized  in  order  to  explore  the  feasibility  and  utility  of  measures 
that  limit  demand  on  LLC  users  and  stakeholders. 

The  third  objective  of  this  research  was  to  explore  the  requirements  of  LLC  self- 
assessment.  This  exploration  was  to  include  an  investigation  of  the  feasibility  of  automated  data 
collection  as  well  as  the  capabilities  and  number  of  required  personnel  and  technologies. 
Requirements  and  methods  for  conducting  LLC  self-assessment  were  determined  via  a 
combination  of  interviews,  literature  review,  and  examination  of  LLC  assessment  lessons 
learned. 

This  final  report  documents  the  modifications  made  to  the  LLC  Assessment  Framework 
and  presents  the  completely  revised  framework  in  Appendix  B.  Next,  the  assessment  of  the  Fort 
Gordon  LLC  is  presented,  detailing  the  method  and  findings.  Finally,  recommendations  for 
enabling  LLC  self-assessment  and  for  conducting  future  LLC  assessment  research  are  provided. 
For  readability  purposes,  this  report  was  written  in  a  modular  format  such  that  the  text  on  each 
area  covered  (Assessment  Framework  Modifications,  Fort  Gordon  LLC  Assessment,  and  LLC 
Self-Assessment)  may  be  read  as  a  stand-alone  document.  The  Conclusions  section  at  the  end  of 
this  report  summarizes  the  report  and  makes  recommendations  for  strengthening  LLC 
assessment  research. 

LLC  ASSESSMENT  FRAMEWORK  MODIFICATIONS 

There  are  several  differences  between  MOS-based  LLCs  and  LLCs  based  on  leader 
education  that  have  implications  for  how  each  type  of  LLC  should  be  assessed.  Every  attempt  to 
maximize  the  generalizability  of  the  LLC  Assessment  Framework  was  made  in  the  previous  ARI 
research  by  including  program-level  stakeholders  in  the  design  process  as  well  as  high-level 
representatives  of  LLCs  other  than  the  one  at  Fort  Leavenworth.  An  in-depth  examination  of  the 
Fort  Gordon  LLC  was  conducted  to  put  the  generalizability  of  the  assessment  framework  to  the 
test.  The  Fort  Gordon  LLC  was  selected  because  it  was  the  longest  running  LLC  and  because  it 
provided  a  large  variety  of  instruction  that  could  be  considered  representative  of  the  types  of 
instruction  most  LLCs  would  offer  (e.g.,  MOSQ  instruction,  simulation  downloads,  on-demand 
training,  leader  education,  etc.).  In  this  section,  a  brief  review  of  the  original  LLC  Assessment 
Framework  is  provided,  followed  by  an  overview  of  the  Fort  Gordon  LLC  and  a  detailed 
discussion  of  the  revisions  necessary  to  the  original  assessment  framework  based  on  the  present 
research. 
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The  Original  LLC  Assessment  Framework 

The  design  of  the  original  LLC  Assessment  Framework  was  based  on  the  logic  model 
approach  to  conducting  program  evaluation  (McLaughlin  &  Jordan,  2004).  Logic  modeling  links 
resources  invested  in  a  program  to  organizational  impact  through  staff  and  stakeholder  activities, 
program  outputs,  and  anticipated  outcomes  at  the  individual  and  small  group  or  community 
level.  A  logic  model  can  be  thought  of  as  a  “high-payoff  target  list”  for  conducting  assessment. 
That  is,  the  elements  of  a  logic  model  are  those  assessment  targets  that  must  be  captured  in  order 
to  conduct  an  informative,  diagnostic  program  evaluation.  A  logic  model  also  may  be  thought  of 
as  a  hypothesis  or  “qualitative  causal  model”  of  how  a  program  achieves  impact.  Assessment 
results  at  the  end  of  the  causal  chain  (i.e.,  impact)  may  be  explained  by  findings  further  up  along 
the  causal  chain  (e.g.,  effectiveness  of  staff  activities,  productivity).  Logic  models  are  commonly 
used  for  assessing  programs  that  do  not  have  simple  retum-on-investment  metrics,  such  as  social 
service  interventions  (e.g.,  parenting  classes,  health  literacy  initiatives),  and  may  readily  be 
adopted  for  non-profit  educational  initiatives. 
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Figure  1 .  Logic  Model  for  LLC  Impact  (from  Cianciolo,  2007) 
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Figure  1  above  shows  the  logic  model  presented  in  Cianciolo  (2007).  This  logic  model 
served  as  the  architecture  for  the  original  LLC  Assessment  Framework.  Briefly,  resources  were 
the  investment  of  money,  labor,  facilities,  and  technology  into  an  LLC.  Activities  were  the  use  of 
resources  by  personnel  to  enable  anytime,  anywhere  access  to  proponent  learning  content.  The 
output  of  an  LLC  was  represented  by  the  access  to  and  use  of  the  portal  and  its  contents  by 
Soldiers.  Short-term ,  intermediate-term,  and  long-term  outcomes  represented  the  instructional, 
individual,  and  organizational  benefits  achieved  by  providing  anytime,  anywhere  access  to 
proponent  learning  content.  For  a  detailed  breakdown  of  the  logic  model  components  into  their 
constituent  elements  the  reader  is  referred  to  Cianciolo  (2007). 

Overview  of  the  Fort  Gordon  LLC 

The  lifelong  learning  concept  emerged  from  the  conditions  faced  by  signal  Soldiers  and 
leaders  working  in  the  contemporary  operating  environment.  Specifically,  the  increased 
importance  of  communications  personnel  to  mission  success — combined  with  rapid  changes  to 
communication  technologies  and  the  side-by-side  use  of  new  and  legacy  systems — created  a 
demand  for  more  frequent,  more  rapid  training  (Farrell,  2001).  For  highly  technical  specialties, 
access  to  formal  training  not  only  was  a  function  of  room  in  the  schoolhouse  but  also  of  the 
availability  of  equipment  for  hands-on  training.  Training  requirements  quickly  outpaced  the  U.S. 
Army  Signal  Center’s  ability  to  provide  in-house  instruction,  so  a  new  way  of  doing  business 
was  required  to  meet  the  need.  The  lifelong  learning  concept  provided  this  new  business  model 
(Wilson  &  Helms,  2003).  The  broad  range  of  training  and  education  strategies  enabled  by  the 
Fort  Gordon  LLC  represents  a  majority  of  the  learning  strategies  adopted  by  other  current  and 
future  LLCs,  making  the  Fort  Gordon  LLC  a  useful  testbed  for  evaluating  the  generalizability  of 
the  LLC  Assessment  Framework. 

The  Fort  Gordon  LLC  was  established  in  March,  2002  by  the  Signal  Center  Directorate 
of  Training  to  implement  the  lifelong  learning  concept  (Walton,  2003).  The  University  of 
Information  Technology  (UIT),  as  the  LLC  was  then  known,  began  by  providing  a  collaborative 
space  and  a  variety  of  training  products  (e.g.,  equipment  simulations)  online  via  a  central 
resource  center  or  portal.  This  resource  center  enabled  anytime,  anywhere  access  to  proponent 
schoolhouse  content  through  individual  workstations  or  the  establishment  of  virtual  campuses 
located  throughout  the  world. 

The  early  success  of  the  UIT  led  to  the  Army-wide  adoption  of  the  lifelong  learning 
concept  (TRADOC,  2004),  with  the  UIT  serving  as  the  first  and  model  instantiation  and 
becoming  known  as  the  Fort  Gordon  LLC.  Additional  pilot  LLCs  were  launched  at  the 
Command  and  General  Staff  College  at  Fort  Leavenworth  and  the  Maneuver  Support  Center  at 
Fort  Leonard  Wood,  among  other  locations,  as  part  of  a  TRADOC  program  to  stand  up  LLCs  at 
every  proponent  schoolhouse. 

In  late  2006  and  in  2007,  the  mission  of  the  Fort  Gordon  LLC  expanded  significantly, 
and  has  been  folded  into  the  larger  LandWarNet  eUniversity  initiative.  LandWarNet  is  the  Army 
component  of  the  Global  Information  Grid,  providing  Army-wide  on-demand  information 
collection,  processing,  storage,  and  dissemination.  It  consists  of  all  of  the  Army’s  Department  of 
Defense/Joint  communications  and  computing  systems  and  services,  software,  data  security 
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services,  and  other  associated  services.  The  LandWarNet  eUniversity,  which  is  supported  by  Fort 
Gordon  LLC  staff,  equipment,  and  affiliated  personnel  (e.g.,  training  content  developers), 
provides  essential  LandWarNet  network  training  and  knowledge  management  to  non-signal 
Soldiers  and  to  civilians  who  touch  the  network.  Through  the  LandWarNet  eUniversity  the  Fort 
Gordon  LLC  has  an  Army-wide  impact  on  information  technology  education  and  training. 

To  manage  the  scope  of  the  present  analysis  and  to  maintain  consistency  with  previous 
research  (i.e.,  Cianciolo,  2007),  the  LandWarNet  eUniversity  was  not  examined  as  a  whole. 
Rather,  focus  was  maintained  on  those  aspects  of  LandWarNet  eUniversity  that  reflected  the 
generic  LLC  model  adopted  by  other  proponents.  Each  of  these  aspects,  which  represent  the 
outputs  of  resource  investment  and  staff  activities,  is  described  in  detail  below. 

LandWarNet  eUniversity  -  Signal 

The  LandWarNet  eUniversity-Signal  (eSignal)  portal  is  the  most  direct  reflection  of  the 
generic  LLC  model.  That  is,  it  features  Blackboard™  as  a  central  element  for  hosting 
standardized,  approved  proponent  curriculum  materials  in  a  web-based  environment  that  is 
accessible  to  learners  (regardless  of  location)  24  hours  a  day,  7  days  a  week.  Also  consistent  with 
the  generic  LLC  model,  eSignal  provides  one-stop  access  to  online  discussion  forums.  eSignal 
differs  significantly  from  other  LLCs  in  the  nature  of  its  instructional  content  and,  by  extension, 
its  expected  impact  on  mission  readiness. 

Military  operational  specialty  qualification  (MOSQ)  instruction.  In  contrast  to  more 
“university-based”  LLCs,  such  as  the  Fort  Leavenworth  LLC,  which  focus  on  leader  education, 
eSignal  primarily  hosts  military  operational  specialty  qualification  (MOSQ)  instruction.  As  of 
the  writing  of  this  report,  qualification  courses  for  20  MOS  are  hosted  on  Blackboard  for  learners 
in  residence  at  Fort  Gordon.  eSignal  also  makes  MOSQ  instruction  available  to  the  Total  Army. 
The  qualification  course  for  one  MOS  (25B10:  Information  Systems  Operator  -  Analyst)  is 
hosted  in  Blackboard  for  students  in  the  Army  Reserves,  with  167  graduates  to  date.  Learners  in 
the  Reserve  Component  participate  in  classroom-based  MOSQ  instruction  at  Regional  High- 
Tech  centers  located  in  Sacramento,  CA  and  Tobyhanna,  PA.  Additionally,  a  25B10  pilot 
blended-leaming  course  for  Army  National  Guard  Soldiers  was  conducted  in  2005.  Distance 
learning  was  provided  to  individual  students  and  an  abridged  classroom  component  was  held  at 
the  Professional  Education  Center  located  in  Little  Rock,  AR. 

Anytime,  anywhere  MOSQ  instruction  is  expected  to  have  a  more  direct  impact  on 
readiness  than  leader  education  for  multiple  reasons.  First,  MOSQ  training  addresses  specific 
skills  that  constitute  effective  individual  and  collective  performance  in  the  field.  Absence  of 
these  skills  affects  readiness  by  reducing  the  number  of  Soldiers  in  a  unit  who  can  carry  out  the 
unit’s  mission  essential  tasks.  The  absence  of  leader  skills  certainly  impairs  readiness,  but  the 
link  between  formal  leader  education  and  leader  effectiveness  in  the  field  is  subject  to  a  greater 
number  of  moderating  factors  [i.e.,  acquisition  and  use  of  experience-based,  or  tacit  knowledge 
(Tan  &  Libby,  1997;  Wagner  &  Sternberg,  1985)], 

Second,  MOSQ  instruction  available  “anywhere”  enables  Soldiers  to  learn  independently 
of  seats  in  the  schoolhouse.  Backlogs  of  students  awaiting  reserved  seats  in  the  schoolhouse 
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reduce  readiness  by  decreasing  the  percentage  of  personnel  in  units  who  are  qualified  to  perform 
their  jobs  (Cianciolo,  2007;  Shanley,  Leonard,  &  Winkler,  2001).  Moreover,  personnel  who  are 
not  available  to  the  unit  because  they  are  attending  schoolhouse  instruction  (or  because  they  are 
not  yet  MOS-qualified)  reduce  the  training  and  mission  readiness  of  the  unit  (Leonard  et  al., 
2001).  Such  absences  are  a  particular  problem  for  the  Reserve  Component,  which  consistently 
has  difficulty  reaching  optimal  training  status  (Sortor,  Lippiatt,  Polich,  &  Crowley,  1994). 
Making  MOSQ  training  available  anywhere  enables  Soldiers  to  leam  from  home,  from  local 
technical  centers,  or  from  deployed  locations,  thus  avoiding  long-term  absences  from  the  unit. 

The  resident  learning  process  may  also  benefit  from  anytime,  anywhere  education  by 
enabling  students  to  take  greater  responsibility  for  their  own  learning,  to  prepare  better  for 
classroom  discussion  and  exercises,  and  to  process  course  information  in  greater  depth. 
Improvements  in  learning  processes  enabled  by  advanced  technologies  are  expected  to  enhance 
readiness  indirectly  over  the  longer  term,  and  have  been  discussed  previously  (Cianciolo,  2007), 
so  they  are  not  further  elaborated  here. 

Simulations.  Providing  anytime,  anywhere  access  to  MOSQ  instruction  is  not  the  only 
means  by  which  eSignal  (and  other  MOS-based  LLCs)  can  have  a  direct  impact  on  readiness. 
Equipment  simulations  hosted  in  eSignal  support  MOS  sustainment  and  refresher  training  as  well 
as  just-in-time  skill  development.  eSignal  hosts  several  equipment  simulations  (e.g.,  AN/TSC- 
85/93,  Joint  Network  Node,  FBCB2,  etc.),  which  are  available  as  downloads  to  registered  users 
who  have  common  access  cards.  These  simulations  are  enabled  by  advanced  interactive 
multimedia  instruction  technologies  that  use  a  scaffolding  approach  to  building  procedural 
technical  skills  (Frank,  G.,  Whiteford,  B.,  Hubal,  R.,  Sonker,  P.,  Perkins,  K.,  Arnold,  P.,  et  al., 
2004).  The  simulations  first  assist  learners  in  acquiring  a  skill  by  demonstrating  each  step  of  the 
procedural  task  along  with  text-based  descriptions  of  the  task.  Next,  learners  practice  the  skill, 
receiving  corrective  feedback  from  the  simulation  when  they  make  mistakes.  Finally,  learners 
validate  their  skills  by  executing  a  task  in  the  simulation  without  feedback.  The  simulations 
produce  a  GO/NOGO  report  for  learners  to  self-evaluate  their  proficiency  (Frank  et  al.,  2004). 

Technical  discussion  forums.  eSignal  also  hosts  discussion  forums  to  support  rapid, 
horizontal  information  and  knowledge  sharing  among  communication  professionals.  The  long¬ 
term  vision  for  eSignal  knowledge  management  is  that  it  will  serve  as  one-stop  access  to  all 
communication-related  discussion  forums.  As  of  1  October  2007,  eSignal  hosts  40  technical 
forums  (with  841  topics),  in  which  signal  Soldiers  share  information  related  to  technical 
troubleshooting  of  the  eSignal  website,  all  aspects  of  communications  equipment  (e.g.,  setup, 
use,  troubleshooting,  maintenance,  lessons  learned,  etc.),  and  the  larger  signal  profession. 

LandWarNet  Leaders  Forum 

Currently  supporting  the  Fort  Gordon  LLC’s  knowledge  management  component  is  the 
Battle  Command  Knowledge  System  (BCKS).  BCKS  hosts  and  manages  the  LandWarNet 
Leaders  Forum ,  which  is  used  by  the  Signal  Center  to  provide  early  exposure  to  knowledge 
management  and  the  use  of  BCKS  during  the  Basic  Officer  Leadership  Course  (BOLC),  the 
Basic  Non-Commissioned  Officer’s  Course  (BNCOC),  and  the  Advanced  Non-Commissioned 
Officer’s  Course  (ANCOC).  The  intent  behind  the  Leaders  Forum  is  to  facilitate  the 
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development  of  signal  Soldiers  into  active  participants  in  horizontal  knowledge  sharing  and 
professional  community  growth  throughout  their  careers.  The  Leaders  Forum  also  provides  easy 
access  to  other  BCKS  professional  forums,  including  NCO  Net,  PlatoonLeader  and 
CompanyCommand. 

The  Leaders  Forum  is  not  directly  accessible  through  the  eSignal  portal.  It  is  accessed 
through  BCKS  via  Army  Knowledge  Online.  Future  plans  include  combining  the  eSignal  and 
BCKS  forums  in  order  to  provide  one-stop  knowledge  management.  In  addition,  external 
communications-related  forums,  to  include  the  signal  Warrant  Officers  ’  forum  and  the  53 
Listserve  (for  automation  personnel),  will  be  folded  into  the  eSignal  site.  These  plans  are 
consistent  with  the  recent  integration  of  the  Army’s  Directorate  of  Information  Management 
portal  for  training,  education,  and  knowledge  sharing  into  the  larger  LandWarNet  eUniversity. 

On-Demand  Training 

On-Demand  Training,  enabled  by  online  courseware  delivery,  makes  MOSQ, 
sustainment,  refresher,  and  just-in-time  training  available  to  signal  units  in  the  field.  On-Demand 
Training  may  take  the  form  of  (1)  mobile  training  teams  (MTTs),  which  travel  from  the  Signal 
Center  to  present  tailored  classroom  instruction  at  remote  locations  by  accessing  content  posted 
in  the  LLC;  (2)  virtual  mobile  training  teams,  which  deliver  special  purpose  instruction  through 
the  LLC  to  deployed  units;  and  (3)  unit  universities,  which  enable  remote,  unit-based  classroom 
instruction  by  accessing  content  posted  in  the  LLC1. 

Importantly,  On-Demand  Training  makes  it  possible  to  provide  training  where  previously 
it  was  impossible  or  at  least  very  difficult.  The  24/7  availability  of  On-Demand  Training  enables 
Soldiers  to  develop  critical  skills  independently  of  the  institutional  education  cycle.  Particularly 
in  the  case  of  technical  skills,  the  requirement  to  enhance  or  refresh  a  Soldier’s  capability  occurs 
several  times  between  scheduled  educational  milestones.  For  instance,  the  refresh  rate  for 
communications  equipment  is  approximately  1 8  months,  but  the  span  between  formal  training 
opportunities  in  the  schoolhouse  typically  is  six  years.  In  this  way,  On-Demand  Training 
supports  the  Army’s  apprenticeship  model  of  training,  in  which  the  majority  of  learning  occurs 
on  the  job.  On-Demand  Training  provides  the  mentor  to  which  Soldiers  are  apprenticed  with  up- 
to-date  training  content,  advanced  technology,  and  sound  pedagogy. 

As  of  the  writing  of  this  report,  one  MTT  from  Fort  Gordon  has  been  deployed  to  Fort 
Hood  to  conduct  post-deployment  25U30  MOSQ  instruction.  One  virtual  mobile  training  team 
was  assembled  to  assist  a  unit  in  Iraq  with  tearing  down,  moving,  and  setting  up  a  new  piece  of 
satellite  communications  equipment.  The  short-course  (videotaped  demonstrations  with 
instructors  and  manufacturer  operation,  equipment,  and  repair  manuals)  took  the  LLC  staff  two 
weeks  to  prepare  with  an  estimated  cost  savings  of  greater  than  $400, 0002.  Forty-seven  Unit 


1  Other  components  of  the  Fort  Gordon  LLC,  including  simulation  downloads,  the  technical  discussion  forums,  and 
non-resident  25B10  qualification  instruction,  may  be  considered  on-demand  training  due  to  their  independence  from 
the  schoolhouse.  The  distinction  between  these  outputs  of  the  LLC  is  made  to  clarify  the  discussion  and  later 
analyses  of  functionality  and  effectiveness. 

2  Estimated  cost  savings  were  presented  by  the  Fort  Gordon  LLC  in  a  VIP  Brief  on  the  Fort  Gordon  LLC,  dated  July 
2007. 
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Universities  have  been  established,  up  from  seven  in  November  of  2006.  Ultimately,  Unit 
Universities  will  be  made  available  to  all  types  of  unit  that  have  a  signal  element,  which  could 
number  more  than  600. 


Mobile  training  teams  (virtual  or  otherwise)  are  activated  at  the  request  and  expense  of  a 
unit.  To  establish  a  Unit  University,  a  unit  commander  or  training  manager  (e.g.,  the  battalion  S3 
or  training  NCO)  contacts  the  LandWarNet  eUniversity  staff  and  requests  access  to  particular 
MOS  course  material.  If  the  requested  MOS  courseware  is  not  among  the  20  MOS  course 
curricula  already  available  on  the  LandWarNet  eUniversity  server,  the  LandWarNet  staff 
procures  the  training  by  contacting  the  instructors  who  deliver  the  course  at  the  Signal  Center. 
Placing  the  courseware  onto  a  website  created  for  the  requesting  unit  launches  the  Unit 
University  with  the  sustainment  training  serving  as  the  initial  focus.  Once  a  Unit  University  is 
established,  users  also  have  one-stop  access  to  other  features  of  the  LandWarNet  eUniversity, 
such  as  equipment  simulation  downloads. 


Modifications  to  Logic  Model  Components 


Few  modifications  to  the  underlying  logic  model  components  of  the  LLC  Assessment 
Framework  were  anticipated  because  the  logic  model  used  to  create  the  framework  was  theory- 
based  and  designed  to  generalize  across  technology-assisted  educational  initiatives,  both  civilian 
and  military  (see  Cianciolo,  2007).  It  was  discovered  through  analysis  of  the  Fort  Gordon  LLC, 
however,  that  some  refinement  to  the  logic  model  and  its  constituent  elements  was  necessary  to 
represent  a  more  complete  hypothesis  of  cause  and  effect  with  regard  to  LLC  impact.  The  main 
drivers  of  change  were  (1 )  the  realization  that  the  timeframe  of  impact  (short-term,  intermediate- 
term,  and  long-term)  was  unrelated  to  the  breadth  of  impact  (individual,  unit,  organization);  and 
(2)  the  discovery  of  external  factors  that  moderate  the  link  between  LLC  outputs  and  the 
expected  outcomes.  The  revised  LLC  logic  model  is  shown  below  in  Figure  2. 


Resources 
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Activities 
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Outcomes  Unit  Outcomes 
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Figure  2.  Revised  LLC  Logic  Model 
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Timeframe  of  Impact 

The  Fort  Gordon  LLC’s  delivery  of  training  outside  the  context  of  formal  educational 
milestones  (e.g.,  training  on  demand,  simulation  downloads),  makes  it  possible  to  achieve 
individual  outcomes  during  the  short-term  that  were  originally  thought  to  occur  during  the 
intermediate-term  (e.g.,  enhanced  readiness).  Outcomes  thought  to  occur  at  the  organizational 
level,  over  the  long-term,  may  actually  occur  at  the  individual  level  (e.g.,  culture  shift)  or  over 
the  short-term  (e.g.,  cost  savings)  as  opportunities  to  provide  just-in-time  training  arise.  Given 
the  generally  limited  scope  of  program  assessments  and  the  constantly  changing  nature  of  Army 
initiatives  such  as  the  Fort  Gordon  LLC,  considering  outcomes  in  terms  of  the  timing  of  impact 
may  draw  attention  to  assessment  targets  that  are  not  observable  or  particularly  meaningful. 

A  more  apt  categorization  of  outcomes  discriminates  between  individual,  unit,  and 
organizational  outcomes,  rather  than  short-term,  intermediate-term,  and  long-term  outcomes. 

•  Individual  outcomes  are  those  changes  in  learner  behavior  and  capability  accomplished 
via  technology-assisted  instruction,  training  on  demand,  and  associated  enhancements  to 
instructional  efficiency  and  effectiveness. 

•  Unit  outcomes  are  those  changes  in  unit  effectiveness  enabled  by  anytime,  anywhere 
access  to  proponent  learning  content. 

•  Organizational  outcomes  are  those  changes  to  organizational  functioning  at  the 
classroom,  proponent,  and  Army  level  accomplished  via  lifelong  learning. 

External  Factors 

As  the  diversity  of  an  LLC’s  educational  offerings  increases,  and  the  breadth  of  outreach 
expands,  the  opportunity  arises  for  external  factors  outside  the  direct  sphere  of  influence  of  the 
LLC  to  play  a  role  in  achieving  impact.  In  these  cases,  organizational  readiness  factors,  which 
are  commonly  recognized  to  influence  the  effectiveness  of  educational  initiatives  (e.g..  Dean, 
Biner,  &  Coenen,  1996;  Salas,  Rhodenizer,  &  Bowers,  2000;  Leonard,  Winkler,  Hove,  et  al., 
2001),  become  even  more  important  determinants  of  success.  The  requirement  for  coordination 
with  external  actors  is  especially  strong  in  a  case  such  as  the  Fort  Gordon  LLC,  where  impact 
depends  on  the  actions  of  others  who  do  not  work  within  the  same  hierarchy  or  reward 
structures.  Examples  of  external  factors  influencing  LLC  effectiveness  include  (but  are  not 
limited  to)  (1)  the  duration  of  the  institutional  curriculum  development  cycle;  (2)  the  availability 
of  computing  facilities  located  on  post  to  support  blended  resident  instruction;  (3)  the  incentive 
program  for  engaging  in  required  at-home  learning;  and  (4)  Army  policy  regarding  who  can 
teach  what  qualification  courses.  The  actors  who  could  influence  these  factors  range  widely  from 
unit  commanders,  to  human  resource  managers,  to  Army-level  plans  and  operations  personnel. 

External  factors  were  not  represented  in  the  original  logic  model  because  they  were 
considered  outside  of  the  sphere  of  influence  of  the  LLC  initiative.  However,  they  do  play  a 
critical  role  in  enabling  success  and  therefore  should  be  recognized  in  an  LLC  assessment  effort. 
Failure  to  account  for  external  factors  would  prevent  a  comprehensive  understanding  at  the 
program  and  Army  level  of  how  resources  must  be  leveraged  or  processes  enhanced  to  maximize 
impact.  As  an  analogy,  the  lifelong  learning  concept  has  been  considered  in  this  research 
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program  to  be  the  main  effort,  but  its  success  is  determined  by  the  actions  of  its  supporting 
efforts.  If  the  commander  is  unaware  of  how  supporting  effort  activity  influences  the  success  of 
the  main  effort,  he  lacks  the  critical  information  he  needs  to  ensure  mission  success. 

Table  1  below  presents  the  revised  logic  model  elements  list.  As  with  the  original  LLC 
logic  model,  each  element  is  tied  to  an  element  category  and  a  logic  model  component. 


Table  1.  Components  and  Associated  Elements  in  the  Revised  LLC  Logic  Model 


Resources 

Money 

o  Costs  directly  and  indirectly  attributable  to  delivering  proponent  courses  (resident,  non¬ 
resident,  and  simulation-supported),  including  technical  staff  costs 
o  Costs  directly  and  indirectly  attributable  to  delivering  training  on  demand 
o  Costs  directly  and  indirectly  attributable  to  implementing  discussion  forums 

Personnel 

o  Leadership,  technical  staff,  course  developers,  courseware  producers,  and  instructors 

Fixed  Assets 

o  Technology,  equipment,  supplies,  and  facilities  used  to  develop,  implement,  and  maintain 

LLCs  and  to  provide  proponent  learning  content 

Activities 

Technical  Staff 

o  Setup,  integrate,  customize,  and  manage  LLC  components 
o  Migrate  course  content  across  LLC  components 

o  Provide  training  to  instructors,  curriculum  developers,  and  other  users  on  the  LLC 
components 

o  Provide  technical  support  to  students,  faculty,  curriculum  developers,  and  other  users 
o  Provide  support  for  answering  Field  Army  users’  operational  questions 

Curriculum  Developers 

o  Collaboratively  generate  course  content  using  LLC  applications 

o  Lead  the  development  of  standard  operating  procedures  (SOPs)  for  leveraging  the  capabilities 
of  the  LLC  components 

o  Mentor  late  adopters  on  system  functionalities  to  enhance  course  development 

Instructors 

o  Deliver  curriculum  materials  by  posting  them  in  the  LLC 

o  Customize  course  content  based  on  student  feedback  and  access  to  other  resources  (e.g.,  Army 
Knowledge  Management) 
o  Perform  course  administrative  duties 

o  Evaluate  student  progress  and  report  grades  to  school  administrators 

o  Lead  the  development  of  SOPs  for  leveraging  system  capabilities 

o  Mentor  late  adopters  on  application  functionalities  to  enhance  course  instruction 

CBT/WBT  Courseware  Production  Team 

o  Maintain  project  teams  to  perform  CBT/WBT  analysis,  design,  development,  implementation, 
maintenance,  and  validation 

o  Provide  contractual,  technical,  and  educational/quality  oversight  of  contractor-developed 
CBT/WBT 

o  Maintain  a  database  of  CBT/WBT  technologies,  capabilities,  and  techniques 

Leadership 

o  Provide  and  communicate  vision 

o  Initiate  and  oversee  user  and  stakeholder  needs  assessment 
o  Market  the  LLC  concept  to  stakeholders 

o  Procure  resources  to  maintain/update  the  LLC,  oversee  operations,  prioritize  limited  resources 
across  LLC  functions 

o  Initiate/organize  the  development  of  SOPs  for  leveraging  system  capabilities 

Outputs 

24/7  Uniform  Access 
o  Access  to  the  system 
o  Actual  use  of  the  system 
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Comouter-/Web-based  courseware  for  delivering  proponent  courses 

o  Courseware  that  is  readily  available  to  meet  course  needs 

External 

Factors 

Culture 

o  Unit  commander  support  of/emphasis  on  informal  education 

Resources 

o  Time  available  for  learners  to  conduct  studies  in  the  context  of  other  work 
o  Fiscal  incentives  available  for  learners  to  conduct  studies  at  home 
o  Technology  and  procedures  available  to  track  individual  learners 
o  Contracting  cycle 
o  Computing  facilities 

Policies 

o  Army-level  policy  supporting  anytime/anywhere  learning 
o  Army-level  policy  supporting  rapid  course  updates 

Outcomes 

(Individual) 

Improved  Student  Performance 

o  Enhanced  higher-order  thinking 
o  Enhanced  skill  development 
o  Enhanced  reflective  capability 
o  Enhanced  learner  independence  and  responsibility 
o  Enhanced  learning  self-efficacy 
o  Enhanced  motivation 

Adoption  of  Lifelong  Learning  Orientation 
o  Distal  motivation  to  engage  in  opportunities  to  learn 
o  Enhanced  collaboration  orientation 
o  Internalization  of  anytime,  anywhere  learning 

Enhanced  Mission  Readiness 

o  Just-in-time  competency 
o  Enhanced  skill  retention 

o  Enhanced  (affective)  organizational  commitment 
o  Enhanced  socialization  in  organizational  goals  and  values 
o  Reduction  in  work-education-Family  conflict 

Outcomes 

(Unit) 

Enhanced  Mission  Readiness 
o  Enhanced  unit  status  reporting 
o  Reduced  time  to  optimal  training  status 
o  Enhanced  MOSQ  training  status 
o  Enhanced  collective  training 

Outcomes 

(Organization) 

Improved  Teaching  and  Learning  Environment 

o  Enhanced  relevance  of  training  and  educational  content 
o  Enhanced  instructional  efficiency 
o  Instructor  as  a  facilitator  of  adult  learning 
o  Presence  of  a  learning  community 

o  Advanced  CBT/WBT  courseware  for  distributed/distance  learning 

Enhanced  Educational  Cost-Effectiveness 

o  Enhanced  cost-outreach 
o  Enhanced  throughput  effectiveness 
o  Reduced  recycle  rate 

o  Enhanced  CBT/WBT  courseware  development  cost-effectiveness 
o  Reduced  the  range  equipment/supplies  requirements 

Modifications  to  Metrics 

As  with  the  LLC  logic  model,  relatively  little  modification  to  metrics  was  anticipated. 
Where  modifications  were  anticipated,  they  were  expected  to  involve  additions  to  outcomes  in 
order  to  reflect  the  unique  ways  in  which  MOS-based  LLCs  enhance  readiness  at  the  unit  level. 
In  fact,  several  metrics  were  added.  In  general,  metrics  were  added  in  order  to  better  represent 
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the  resources,  activities,  outputs,  and  outcomes  associated  with  providing  learning  opportunities 
other  than  course  instruction  (i.e.,  simulation  downloads,  discussion  forums,  and  training  on 
demand).  Although  the  original  LLC  Assessment  Framework  accounted  for  the  different  types  of 
learner  reached  by  the  Fort  Leavenworth  and  Fort  Gordon  LLCs,  the  framework  did  not  account 
for  the  diverse  range  of  instructional  methods  possible.  The  complete  set  of  metrics  for  the 
revised  LLC  Assessment  Framework  is  shown  in  Appendix  B. 

Modifications  to  Measures 

Some  modification  to  measures  was  anticipated.  Primarily,  it  was  expected  that 
modifications  would  include  (1)  the  addition  of  measurement  methods  that  could  serve  as 
feasible  alternatives  to  surveys;  (2)  the  modification  of  existing  measures  to  make  them  more 
general  across  LLCs  (i.e.,  not  tied  to  a  specific  mode  of  course  delivery  or  type  of  curriculum); 

(3)  the  addition  of  measures  to  address  the  broader  range  of  assessment  opportunities  enabled  by 
the  diverse  outputs  of  different  types  of  LLC;  and  (4)  the  addition  of  measures  to  address  the 
creation  of  new  metrics,  where  applicable.  All  of  these  anticipated  modifications  were  made. 

First,  interviews  and  focus  groups  were  added  as  feasible  alternatives  to  surveys.  These 
measurement  methods  provide  significantly  more  diagnostic  information  than  do  surveys  and 
permit  approximately  the  same  sample  representativeness  as  surveys  in  certain  cases  (i.e.,  when 
participants  are  not  located  at  the  schoolhouse),  given  the  relatively  low  response  rate  to  surveys 
by  LLC  users  in  the  field.  The  decision  to  use  surveys  versus  focus  groups  should  be  based  on 
the  diversity  and  accessibility  of  the  population  to  be  sampled,  as  well  as  the  purpose  of  the 
information  (developmental  or  sampling)  to  be  acquired  through  the  research  process. 

Second,  some  measures  were  modified  or  added  to  enable  more  generic  methods  of 
assessing  student  performance  and  more  broadly  applicable  measures  of  resources.  For  instance, 
financial  data  may  not  be  available  in  some  cases,  so  reasonable  methods  for  estimating  costs 
must  be  specified.  The  revised  measures  support  this  estimation  process.  In  addition,  where 
special  purpose  measures  were  listed  explicitly  in  the  original  assessment  framework  (e.g.,  the 
1009  form  used  by  the  Command  and  General  Staff  College  to  assess  higher-order  thinking), 
more  generic  forms  of  assessment  were  supplied.  The  type  of  instruction  to  which  measures 
could  be  applied  also  was  specified  in  a  way  that  would  be  more  user  friendly  and  generalizable 
across  LLCs.  Rather  than  specifying  the  mode  of  instruction  (i.e.,  distributed-collaborative, 
resident,  etc.),  the  appropriate  LLC  output  was  specified  (e.g.,  leader  education,  MOSQ 
instruction,  On-Demand  Training,  etc.).  This  specification  may  help  users  of  the  revised 
framework  to  more  quickly  determine  the  metrics  and  measures  that  apply  to  their  particular 
LLC. 


Third,  archival  materials,  such  as  programs  of  instruction  and  course  crosswalks 
(comparing  resident  to  distance  learning),  were  identified  as  additional  measures  for  several 
metrics.  Such  measures  draw  the  attention  of  assessors  who  are  relatively  new  to  the  Army 
training  system  to  the  possibility  of  cost-effective,  informative  alternatives  to  novel  data 
collection.  Additional  measures  not  explicitly  mentioned  in  the  assessment  framework  include 
previous  scientific  and  institutional  studies.  As  is  demonstrated  in  the  Fort  Gordon  LLC 
assessment  below,  the  analysis  of  previous  research  provided  valuable  supporting  information. 
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Historical  documents  cannot  be  directly  linked  to  any  one  type  of  metric3,  however,  so  they  were 
not  included,  but  should  be  explored  by  future  assessors. 

Finally,  measures  were  added  to  address  the  new  metrics.  Refer  to  Appendix  B  for  the 
complete,  revised  LLC  Assessment  Framework. 

ASSESSMENT  OF  THE  FORT  GORDON  LLC 

This  section  presents  the  assessment  of  the  Fort  Gordon  LLC.  Six  aspects  of  the  Fort 
Gordon  LLC  were  assessed:  (1)  MOSQ  Instruction;  (2)  Assignment-Oriented  Training;  (3) 
Simulations;  (4)  Discussion  Forums;  (5)  Leader  Education;  and  (6)  On-Demand  Training.  For 
each  aspect  assessed,  the  discussion  is  organized  as  follows: 

•  The  Sample 

•  Assessment  Questions  &  Method 

•  Summary  of  Findings 

•  Detailed  Findings 

•  Recommendations 

The  Fort  Gordon  LLC  is  wide  ranging,  with  20  MOSQ  courses  posted  online,  46  Unit 
Universities,  and  well  over  100  simulation  and  courseware  downloads.  The  limited  scope  and 
duration  of  the  present  assessment  effort  necessitated  that  only  a  sample  of  each  of  the 
components  of  LLC  be  selected  for  analysis.  This  sample  was  chosen  based  on  seniority  (e.g., 
courses  that  were  among  the  first  to  be  hosted  on  the  LLC  were  selected)  and  on  input  from 
project  stakeholders  at  the  Signal  Center  Directorate  of  Training.  In  most  cases,  the  sample 
involved  in  data  collection  was  representative  (if  not  the  population  itself).  Where  the  sample 
may  not  be  representative,  caveats  are  provided.  The  investigation  presented  below  does  not 
feature  controlled  scientific  experimentation,  but  rather  reflects  the  attempt  to  explore  causality 
using  a  qualitative  (logic)  model  in  a  naturalistic  organizational  setting.  Quantitative  data  were 
collected  everywhere  possible  and  presented  in  table  format,  but  both  summarized  and  detailed 
findings  are  provided  in  narrative  format  due  to  the  complex  and  somewhat  qualitative  nature  of 
the  research. 

It  should  be  noted  that  the  Fort  Gordon  LLC  is  rapidly  changing  such  that  the  assessment 
results  presented  here  reflect  a  historical  snapshot  of  LLC  functioning.  For  example,  the 
curriculum  for  the  Signal  Captains’  Career  Course,  assessed  as  part  of  this  investigation,  recently 
underwent  significant  change  in  order  to  standardize  the  content  of  the  Active  and  Reserve 
Component  courses.  The  majority  of  the  Signal  Captains’  Career  Course  students  interviewed 
had  begun  the  course  using  the  previous  format,  completing  it  under  the  new  format.  A  small 
number  of  the  Unit  Universities  initially  selected  for  examination  could  not  be  assessed  because 
the  units  themselves  were  deactivated.  The  Leaders  Forum  staff  acquired  direct  BCKS  assistance 
during  the  course  of  the  research  effort,  and  is  in  the  process  of  developing  near-term  plans  for 


3  The  presence  of  these  documents  is  not  systematic,  although  their  most  likely  focus  is  organizational  impact,  such 
as  cost  savings,  learning  effectiveness,  and  readiness  enhancement.  Most  commonly,  these  studies  are  authored  by 
government  and  federally  funded  research  institutions,  including  the  TRADOC  Analysis  Center,  RAND 
Corporation,  and  ARI. 
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significant  changes  to  the  organization  of  forums.  In  each  section  below,  an  attempt  has  been 
made  to  document  the  implications  of  rapid  change  for  interpreting  the  assessment  findings. 
Continuous  assessment  would  be  necessary  to  ensure  that  the  need  for  and  impact  of  change  is 
tracked. 


Assessment  of  MOSQ  Instruction 


The  Sample  -  25B10 

To  fully  study  the  impact  of  the  Fort  Gordon  LLC  on  MOSQ  instruction,  a  course  had  to 
be  selected  such  that  the  effects  of  the  LLC  on  delivering  and  administering  instruction  could  be 
examined.  As  described  in  Cianciolo  (2007),  educational  technology  may  transform  the 
classroom  experience  both  by  how  it  is  used  to  convey  curriculum  materials  during  class  time 
and  by  how  it  is  used  to  conduct  administrative  activities  outside  the  classroom.  The  lack  of 
computer  lab  facilities  at  Fort  Gordon  prevents  anytime,  anywhere  access  to  course  materials  by 
many  resident  students,  so  the  Fort  Gordon  LLC  currently  is  not  being  used  to  conduct  such 
administrative  activities  as  assigning  or  collecting  homework,  posting  announcements,  or  sharing 
course  content  among  resident  students.  The  non-resident  student  and/or  instructor  experience 
with  accessing  and  using  LLC  technologies  from  a  distance  therefore  was  the  only  such 
experience  that  could  be  examined.  Examining  a  course  that  provides  resident  and  non-resident 
instruction  also  was  necessary  to  determine  whether  the  LLC  enabled  standardization  and 
increased  outreach  of  proponent  curriculum  materials.  Increased  outreach  is  a  critical  mechanism 
by  which  the  cost  per  student  of  a  course  is  reduced.  At  the  time  the  present  research  was 
conducted,  the  25B10  course  was  the  only  MOSQ  course  offered  in  both  resident  and  non¬ 
resident  format  using  the  Fort  Gordon  LLC  technologies  (i.e.,  Blackboard).  For  this  reason,  the 
25B10  course  was  selected  for  analysis  and  a  sample  size  of  one  equals  the  population. 

The  25B10  course  provides  instruction  for  “entry-level”  (Skill  Level  1)  information 
systems  operator-analysts.  The  course  lasts  approximately  19  weeks,  and  is  segmented  into  18 
annexes,  covering  such  topics  as  information  assurance,  network  essentials,  troubleshooting  and 
repair,  routers  and  switches,  basic  operations  of  various  commercial  off-the-shelf  applications 
(e.g.,  Microsoft  Outlook,  UNIX,  and  SOLARIS  operating  systems),  and  basic  operations  of 
proprietary  tactical  equipment  (i.e.,  selected  components  of  the  Army  Battle  Command  System). 
Annexes  comprise  lectures,  demonstrations,  simulations,  and  hands-on  training  in  different 
sequences  depending  on  the  annex.  The  25B10  non-resident  course  is  offered  to  students  in  the 
Reserve  Component  via  High-Tech  Regional  Centers,  where  students  may  come  together  for 
lectures  and  hands-on  training.  Curriculum  materials  (e.g.,  lecture  slides)  also  may  be  accessed 
remotely  by  non-resident  students  via  Blackboard. 

Assessment  Questions  &  Method 

The  analysis  of  MOSQ  instruction  assessed  the  processes  and  outcomes  associated  with 
delivering  the  25B10  course  to  resident  and  non-resident  learners  via  the  Fort  Gordon  LLC. 
Specifically,  the  following  questions  were  asked  and  associated  metrics  used  to  focus  data 
collection: 
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o 


Question:  Does  delivering  25B10  using  Blackboard  enhance  course  relevance? 

o  Metric:  Estimated  frequency  of  instructor  augmentation  to  standardized  course 
curriculum  relative  to  pre-LLC  situations 
o  Question:  Does  using  Blackboard  to  administer  the  25B10  course  in  residence  enhance 
instructional  efficiency? 

o  Metric:  Estimated  %  reduction  in  time  spent  in  classroom  doing  administrative 
tasks,  including  announcements,  handouts,  and  testing  using  LLC 
o  Metric:  Estimated  %  increase  in  time  available  to  assist  students  having 
difficulties 

o  Question:  What  are  the  cost  savings  associated  with  administering  25B 1 0  online? 

o  Metric:  Estimated  %  reduction  in  cost-per-student  with  LLC  versus  prior  to  LLC 
implementation 

o  Metric:  %  reduction  in  travel,  housing,  and  student  pay  expenses 
o  Question:  Does  administering  the  25B10  course  online  enable  uniform  access  to  training 
content? 

o  Metric:  %  curriculum  materials  found  in  Blackboard  that  are  common  across  user 
locations 

The  above  metrics  were  partially  captured  via  a  series  of  interviews  and  a  focus  group. 
Specifically,  the  Division  Chief  of  25B  resident  instruction  was  interviewed,  as  were  the  former 
Fort  Gordon  LLC  Program  Manager,  the  head  instructors  for  the  non-resident  25B10  course 
administered  at  two  Reserve  Component  High-Tech  Regional  Centers  (Tobyhanna,  PA  and 
Sacramento,  CA)  and  at  the  Army  National  Guard  Professional  Education  Center  located  in 
Little  Rock,  AR.  A  focus  group  was  conducted  with  three  resident  25B10  instructors  and  the 
Deputy  Division  Chief  of  25B  resident  instruction.  The  interviews  and  focus  group  were 
conducted  as  an  alternative  to  surveys  for  assessing  instructor  activities  and  impressions. 
Archival  materials  also  were  examined,  including  the  25B10  program  of  instruction,  the  25B10 
non-resident  curriculum  materials,  and  a  cost-savings  analysis  of  a  pilot  non-resident  25B10 
course  conducted  at  the  Army  National  Guard  Professional  Education  Center. 

Summary  of  Findings 

In  the  case  of  25B 1 0,  analysis  of  activities  and  outputs  indicated  that  there  was  great 
potential  for  online  course  delivery  via  the  Fort  Gordon  LLC  to  enhance  the  outreach, 
administration,  standardization,  currency,  and  cost-effectiveness  of  MOSQ  instruction.  The  LLC 
technical  staff  performed  the  activities  required  to  post  curriculum  materials  online,  creating 
three  25B10  Blackboard  sites  that  enabled  students  in  both  the  Reserves  and  National  Guard  to 
earn  MOS-qualification  without  the  obligation  to  leave  their  units  or  families  and  incur  the 
related  travel,  lodging,  and  duty  pay  costs  associated  with  resident  instruction.  Blackboard  sites 
for  the  resident  25B10  course  allowed  instructors  to  present  course  materials  independently  of 
classroom  arrangements.  Moreover,  administering  exams  via  Blackboard  allowed  instructors  to 
reduce  significantly  the  amount  of  time  spent  administering  exams  and  to  use  the  additional  time 
for  remedial  instruction.  The  LLC  help  desk  provided  assistance  to  course  developers  and 
instructors  upon  request. 
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Linking  the  online  delivery  of  25B10  materials  to  fully  realized  benefits,  however,  was 
not  a  straightforward  matter  due  to  factors  outside  the  sphere  of  influence  of  the  Fort  Gordon 
LLC  technical  staff  and  the  Signal  Center  Directorate  of  Training.  These  factors  included  (1)  the 
length  of  the  curriculum  review  and  revision  process  for  programs  of  instruction;  (2)  lack  of 
public  computing  facilities  at  Fort  Gordon;  and  (3)  Army-level  restrictions  on  who  may  teach 
MOSQ  courses. 

The  length  of  the  curriculum  review  process  made  it  very  difficult  for  LLCs  to  facilitate 
meaningful  change  to  institutional  curriculum  content.  Although  some  modification  could  be 
made  without  being  subject  to  the  institutional  review  process,  the  nature  of  such  change  was 
superficial.  Substantive  updates  to  address  new  equipment  or  equipment  versions  could  not  be 
approved  or  resourced  without  formal  review.  Moreover,  instructor  access  to  Blackboard  sites 
was  restricted  such  that  content  edits  or  augmentation  had  to  be  made  through  course  developers. 
This  restriction  ensured  standardization,  but  limited  instructor  participation  in  enhancing  course 
currency.  Lack  of  public  computing  facilities  limited  student  access  to  course  materials  outside 
of  the  classroom,  making  optimal  use  of  class  time  difficult  to  achieve  for  resident  instructors. 
Without  common  access  to  computers  after  hours,  course  administration  had  to  remain  a 
classroom  activity.  Army-level  restrictions  on  who  may  teach  MOSQ  courses  prevented  the 
broad  implementation  of  non-resident  25B10  instruction  hosted  via  the  LLC  and  the  associated 
cost  savings. 

Some  modification  to  internal  processes  would  have  facilitated  the  use  of  Blackboard  to 
enhance  instructional  efficiency  and  to  enable  uniform  access  to  course  content.  Different 
programs  of  instruction  were  used  in  all  three  of  the  25B 10  courses  analyzed  (resident,  non- 
resident/Tobyhanna,  and  non-resident/Sacramento),  in  part  due  to  lags  in  larger  training 
processes  or  schedules  but  also  due  in  part  to  lack  of  coordination  among  the  managers  of 
resident  instruction  and  the  LLC  staff.  Ensuring  standardization  was  not  viewed  as  a  role  for  the 
LLC  staff  (rather,  this  role  was  to  post  content  as  requested)  or  the  resident  instruction  managers 
(rather,  communication  outside  the  schoolhouse  was  handled  by  the  Army’s  Automated  Systems 
Approach  to  Training  personnel).  Similarly,  the  relative  roles  of  the  LLC  staff  and  resident 
instruction  managers  in  ensuring  that  instructors  were  facile  with  technology-assisted  instruction 
and  Blackboard  features  were  undefined.  More  tightly  integrated  roles,  guided  by  a  systems 
perspective  on  producing  educational  change,  were  necessary  to  sustain  the  momentum  of 
proactive  individuals  who  leveraged  the  capability  of  LLC  technologies. 

Detailed  Findings 

Course  relevance.  Army  training  and  education  courses  go  through  a  lengthy  review 
process  prior  to  substantive  changes  being  made  to  the  program  of  instruction.  Substantive 
changes  are  those  changes  that  lead  to  new  requirements  for  resources,  such  as  hands-on  training 
equipment,  simulation-based  courseware,  time,  and  personnel.  Substantive  changes  may  include 
the  introduction/removal  of  a  topic  area  or  the  addition/deletion  of  content  to  an  existing  topic 
area  (such  that  the  required  number  of  hours  for  instruction  changes).  The  resources  necessary  to 
meet  the  new  requirements  are  not  provided  by  the  Training  and  Doctrine  Command  until  the 
revised  program  of  instruction  has  been  reviewed  and  approved.  The  review  process  takes 
approximately  three  years,  a  timeframe  that  lags  behind  communications  equipment 
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modifications.  Because  substantive  changes  to  a  curriculum  are  subject  to  the  institutional  review 
process,  the  LLC  may  help  to  address  this  lag  if  it  is  used  by  instructors  and/or  course  developers 
to  rapidly  make  minor  revisions  to  course  content  or  to  disseminate  new  information  or 
downloads  that  students  could  access  during  time  not  used  for  classroom  instruction. 

The  three  25B10  instructors  and  the  Deputy  Division  Chief  of  25B  resident  instruction, 
who  participated  in  the  focus  group,  indicated  that  approximately  30%  of  a  lesson  plan  may  be 
changed  without  requiring  the  lengthy  institutional  review  process.  These  changes  include 
corrections  to  typos  and  other  errors  in  the  lecture  slides  as  well  as  relatively  minor  revisions  of 
content.  For  example,  slides  could  be  added  to  a  lecture  to  reflect  a  change  in  equipment 
functionality  or  the  presentation  strategies  within  a  lecture  could  be  modified  such  that  a  more 
multimedia  approach  is  used.  Focus  group  participants  reported  that  they  were  not  allowed  to 
make  such  changes  themselves  (i.e.,  they  had  read-only  access  to  Blackboard),  but  were  required 
to  work  with  course  developers  who  have  administrative  access  to  curriculum  materials.  Access 
restrictions  were  built  into  the  curriculum  management  process  to  ensure  the  quality  control  and 
standardization  of  course  materials.  Moreover,  public  access  to  computers  outside  of  the 
classroom  was  not  widely  available  for  resident  students,  so  the  Fort  Gordon  LLC  was  not  used 
to  distribute  extracurricular  materials.  For  these  reasons,  focus  group  participants  did  not  report 
engaging  in  active  modification  of  course  content. 

To  the  extent  that  the  sample  of  instructors  interviewed  was  representative  and  reporting 
was  accurate,  the  Fort  Gordon  LLC  appears  to  have  had  relatively  little  impact  on  enabling 
enhanced  relevance  of  MOSQ  training  content  such  that  MOSQ  instruction  optimally  meets  the 
readiness  requirements  of  field  units.  Online  posting  of  institutional  course  content  can 
strengthen  the  tie  between  the  non-resident  curriculum  and  the  needs  of  the  Army  as  reflected  in 
the  proponent’s  priorities  for  instruction.  However,  the  critical  changes  that  must  be  made  to 
address  the  lag  between  curriculum  revision  and  equipment  updates  are  largely  beyond  the 
sphere  of  influence  of  the  LLC,  requiring  modification  to  the  curriculum  development  and 
resourcing  process  more  so  than  implementation  of  advanced  training  delivery  technologies. 

Such  technologies  are  most  effective  for  standardizing  course  currency  across  resident  and  non¬ 
resident  learners  or  providing  up-to-date  content  that  is  not  tied  to  a  particular  institutional 
course.  This  finding  is  consistent  with  Cianciolo  (2007). 

It  should  be  noted  that  the  adoption  of  new  communications  equipment  by  field  units  also 
lags  behind  the  release  cycles  of  equipment  modifications  or  replacements.  The  educational  and 
readiness  costs  associated  with  curriculum  lag  are  unknown  because  the  degree  of  mismatch 
between  training  content  and  fielded  equipment  is  undocumented.  The  Fort  Gordon  LLC  may 
better  address  readiness  shortfalls  resulting  from  curriculum  lag  by  providing  training  on  demand 
rather  than  by  attempting  to  change  institutional  processes. 

Instructional  efficiency.  Instructional  efficiency  is  enhanced  by  technology  when  it  is 
used  to  reduce  the  amount  of  classroom  time  spent  on  non-teaching  activities  and  increases 
instructor  time  spent  on  tailoring  the  learning  experience  to  the  needs  of  a  particular  class  or 
particular  individuals  (e.g.,  Bourne,  1998).  Focus  group  participants  (N=  3  25B 10  instructors 
and  the  Deputy  Division  Chief  of  25B)  reported  that  there  was  relatively  little  reduction  in 
classroom  administration  time  enabled  by  hosting  25B10  course  content  online  in  Blackboard. 
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The  reason  they  provided  was  that  resident  25B10  students  do  not  have  uniform  access  to 
computing  facilities  outside  of  the  classroom.  They  stated  that  announcements,  homework 
assignments,  and  other  materials  posted  online  would  not  be  accessible  to  most  students,  so  class 
time  was  used  for  course  administration  instead.  In  addition,  they  noted  that  the  25B 1 0  course 
has  very  few  group  assignments,  which  students  use  face-to-face  time  to  complete.  Non-resident 
instructors  at  the  High-Tech  Regional  Centers  reported  not  conducting  online  course 
administration  for  similar  reasons. 

Focus  group  participants  did  indicate,  however,  that  using  Blackboard  could  vastly 
simplify  the  administration  of  quizzes  and  exams,  which  would  in  turn  allow  them  to  make  better 
use  of  class  time.  They  reported  that  students  took  all  quizzes  and  most  exams  in  Blackboard 
with  grading  done  automatically  by  the  system.  Prior  to  administering  assessments  in 
Blackboard,  all  tests  were  paper-based,  requiring  instructors  to  determine  grades  by  hand  and  to 
provide  delayed  feedback  to  students.  Administering  assessments  in  Blackboard  could  enable 
instructors  to  monitor  real-time  trends  in  student  test  performance  such  that  group-level  feedback 
could  be  delivered  immediately  following  evaluation  and  individual  students  experiencing 
problems  could  be  assisted  prior  to  follow-on  instruction.  Automatic  grading  also  could  enable 
instructors  to  spend  less  time  working  at  home  and  more  time  working  with  students  having 
problems  with  the  course  content.  Reducing  the  rate  at  which  failing  students  are  recycled 
through  the  25B10  course  would  reduce  the  housing,  duty  pay,  and  other  costs  associated  with 
spending  extra  time  at  the  schoolhouse  retaking  course  content. 

Focus  group  participants  estimated  that  approximately  20%  of  instructors  currently  were 
able  to  leverage  Blackboard’s  capability  to  administer  exams  as  described  above.  They  also 
reported  that  no  formal  process  existed  for  training  25B10  instructors  to  use  Blackboard  in  this 
way.  One  of  the  two  non-resident  25B10  instructors  interviewed  (representing  50%  of  the 
population  of  such  instructors)  stated  that  a  high  frequency  of  connectivity  problems  altogether 
prevented  the  use  of  Blackboard  for  assessment  administration.  The  other  non-resident  instructor 
interviewed  did  not  report  such  problems,  but  also  did  not  indicate  that  extensive  use  was  made 
of  Blackboard’s  assessment  function  to  enhance  instruction,  perhaps  due  to  staffing  shortfalls. 

In  the  case  of  instructional  efficiency,  as  in  the  case  of  course  relevance,  factors  outside 
the  sphere  of  influence  of  the  Fort  Gordon  LLC  played  a  role  in  limiting  its  impact  on  how 
teaching  and  learning  are  conducted.  Individual  instructors  who  proactively  leveraged 
Blackboard  capabilities  were  able  to  introduce  meaningful  change  to  their  classroom  conduct, 
given  the  limitations  imposed  by  external  factors.  Outreach  by  the  LLC  technical  staff  could 
accelerate  the  process  by  which  other  instructors  gain  proficiency  with  Blackboard’s  grade  book 
features,  thus  helping  them  to  leverage  technology  for  transforming  the  classroom.  The  larger 
matter  of  course  administration  requires  a  shift  in  how  resident  student’s  access  instruction  (and 
how  such  access  is  granted  by  the  authentication  software  on  Army  servers)  such  that  anytime, 
anywhere  access  is  a  meaningful  possibility  for  all  students. 

Cost  savings.  Reducing  the  per-student  cost  of  MOSQ  instruction  may  be  accomplished 
by  ( 1 )  reducing  the  number  of  students  who  must  travel  to  the  schoolhouse  for  instruction;  (2) 
reducing  the  amount  of  time  students  spend  in  the  schoolhouse;  and/or  (3)  increasing  the  number 
of  students  who  participate  in  instruction  (given  roughly  the  same  number  of  instructors, 
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simulations,  classrooms,  etc.)-  Providing  MOSQ  instruction  online  can  reduce  per-student  costs 
along  the  second  of  these  dimensions,  according  to  a  pilot  study  launched  by  the  Signal  Center  in 
July  2004. 


The  purpose  of  the  2004  pilot  study  was  to  explore  the  feasibility  of  providing  25B10 
instruction  to  non-resident  students  in  the  Army  National  Guard.  Prior  to  the  pilot  study, 
National  Guard  students  were  required  to  go  to  Fort  Gordon  to  complete  the  19.5-week  25B10 
course.  There  were  1 7  students  in  the  pilot  non-resident  course,  which  was  administered  as  a 
blend  of  at-home  instruction  and  a  one-week  capstone  exercise  held  at  the  Professional 
Education  Center  in  Little  Rock,  AR.  The  pilot  study  enabled  the  Signal  Center  to  assess  the 
reduction  in  salary  (i.e.,  daily  base  pay),  lodging,  and  meal  expenses  that  would  result  from 
shortening  the  amount  of  time  spent  in  the  schoolhouse  from  19.5  weeks  (137  days)  to  1  week. 
According  to  this  pilot  study,  expenses  were  estimated  as  shown  in  the  following  table. 


Table  2.  Estimated  Daily  Expenses  for  25B10  Instruction  from  the  Signal  Center  Pilot  Study 


Expense  Category 

Amount 

Daily  Base  Pay 

$73.33 

Lodging 

$69.00  (Fort  Gordon,  GA);  $71.00  (Camp  Robinson,  AR) 

Meals 

$39.00 

The  data  in  the  table  indicate  that  the  per-student  duty  pay,  lodging,  and  meals  costs  for 
sending  National  Guard  Soldiers  to  Fort  Gordon  for  the  full  25B10  course  were  $24,842.21  and 
were  $  1 ,283.3 1  for  sending  these  same  students  to  Camp  Robinson  for  the  one- week  capstone 
exercise.  This  represents  a  95%  reduction  in  these  per-student  expenses,  and  a  93%  reduction  in 
cost-per-student  when  travel  expenses  ($550)  are  factored  in.  When  the  other  costs  associated 
with  instruction  (e.g.,  instructor  salaries,  teaching  facilities  upkeep  and  maintenance,  hands-on 
training  equipment,  etc.)  are  considered,  the  %  reduction  in  per-student  cost  will  decrease 
somewhat  further,  but  it  is  unlikely  that  it  will  be  reduced  to  triviality. 

For  example,  if  the  additional  per-student  expenses  associated  with  providing  non¬ 
resident  training  are  estimated  to  be  $25,000,  the  %  reduction  in  cost-per-student  decreases  to 
approximately  50%.  Note,  however,  that  a  $25,000  per-student  cost,  in  addition  to  duty  pay, 
lodging,  meals,  and  travel  expenses,  would  be  extremely  high.  Such  a  per-student  cost  might  be 
incurred  if  (a)  expensive,  sole-purpose  facilities  must  be  built  to  hold  the  non-resident  capstone 
exercise;  (b)  the  increase  in  the  number  of  non-resident  instructors  (and/or  hands-on  training 
equipment)  is  not  matched  by  a  decrease  in  the  number  of  resident  instructors  (and/or 
equipment);  and  (c)  the  courseware  and  simulations  used  for  at-home  instruction  are  very 
expensive  and  have  a  short  useful  life. 

Due  to  Army-level  constraints  on  who  is  allowed  to  provide  25B  MOSQ  instruction,  the 
25B 10  non-resident  course  has  not  been  administered  to  National  Guard  students  since  the  2004 
pilot.  For  this  reason,  additional  cost  savings  associated  with  qualifying  25B 10  National  Guard 
Soldiers  via  non-resident  instruction  have  not  been  achieved.  Analogous  per-student  cost  savings 
may  not  be  seen  for  25B10  students  in  the  Reserve  Component  who  take  the  course  at  High-Tech 
Regional  Centers.  This  is  because  at  High-tech  Regional  Centers,  25B10  students  spend 
approximately  1 3  weeks  of  the  course  in  the  classroom.  It  should  be  noted,  however,  that 
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approval  recently  was  granted  for  the  National  Guard  to  teach  the  25B10  MOSQ  course,  with  the 
necessary  resourcing  to  conduct  the  course  expected  sometime  in  2008.  Reinstatement  of  this 
course  presumably  will  resume  the  associated  cost  savings. 

Uniform  access.  Uniform  access  to  curriculum  materials  includes  the  equivalence  of 
instructional  content  provided  to  resident  and  non-resident  students  as  well  as  the  equivalence  of 
the  learning  experience  (i.e.,  what  is  done  with  instructional  content)  across  both  types  of  learner. 
It  enhances  course  relevance  for  non-resident  learners  in  addition  to  instructional  quality.  Non¬ 
resident  instructors,  as  well  as  the  former  LLC  Program  Manager  and  the  Division  Chief  of  25B 
resident  instruction,  indicated  that  posting  25B 10  materials  online  significantly  enhanced  the 
quality  and  standardization  of  lecture  materials  for  non-resident  students.  They  stated  that  online 
posting  of  course  content  made  lectures  less  dependent  upon  the  individual  strengths  of  the 
instructor  and  more  tightly  tied  to  the  learning  priorities  identified  by  the  proponent.  The  present 
analysis  examined  the  percent  overlap  in  25B10  instructional  materials  made  available  to  non¬ 
resident  learners  and  explored  the  similarities/dissimilarities  in  various  aspects  of  learning 
experience,  including  technology  use  and  classroom  time. 

The  Fort  Gordon  LLC  has  enabled  the  online  posting  of  25B10  curriculum  materials, 
however  the  lead  25B10  instructors  at  the  Tobyhanna  and  Sacramento  High-Tech  Regional 
Centers  reported  that  their  course  was  not  based  on  the  same  program  of  instruction  currently  in 
use  at  the  Signal  Center.  One  of  the  interviewees  reported  that  updates  to  the  resident  25B10 
program  of  instruction  were  not  disseminated  to  the  Reserve  Component  but  that  such  updates 
could  be  acquired  if  one  knew  the  right  person  to  ask.  Both  interviewees  also  reported  that  the 
course  content  available  to  them  through  the  LLC  was  not  the  most  recent  version  developed,  nor 
was  the  recency  of  course  content  consistent  among  them  (i.e.,  they  reported  that  the  High-Tech 
Regional  Center  in  Tobyhanna  was  using  the  2005  version  of  the  courseware  whereas  the  2004 
version  was  being  used  in  Sacramento).  These  claims  were  borne  out  by  examination  of  the  non¬ 
resident  25B10  sites  present  on  the  LLC.  Resident  instructors  indicated  that  lags  in  the  Army’s 
Automated  Systems  Approach  to  Training  process,  which  posts  programs  of  instruction, 
prevented  simultaneous  distribution  of  the  programs  of  instruction.  They  also  reported  that 
different  non-resident  locations  use  different  programs  of  instruction  depending  on  the  resources 
(e.g.,  hands-on  training  equipment)  available  to  them. 

Examination  of  the  25B10  course  syllabus  available  via  the  LLC  revealed  that  both 
Tobyhanna  and  Sacramento  had  the  same  syllabus,  dated  August  2003.  The  contact  information 
present  in  the  syllabus  was  similarly  outdated,  with  the  names  of  technical  support  personnel  and 
key  leaders  in  the  Signal  Center  Directorate  of  Training  no  longer  valid.  The  course  map 
provided  in  the  syllabus  featured  different  annexes  than  those  present  in  either  the  Tobyhanna  or 
Sacramento  Blackboard  sites.  The  annexes  present  in  the  course  websites  reflected  more  recent 
annex  titles,  with  greater  correspondence  (than  those  in  the  syllabus)  to  the  annex  titles  in  the 
2005  resident  25B 10  program  of  instruction  (used  for  2006-07  resident  instruction).  Tobyhanna’s 
annex  titles  had  100%  overlap  with  those  in  the  2005  program  of  instruction  (i.e.,  the  annex 
names  matched  exactly).  Sacramento’s  annex  titles  had  29%  (7/24)  overlap  with  the  2005 
program  of  instruction  and  69%  (9/13)  overlap  with  the  2004  program  of  instruction.  A  more 
recent  (2006)  resident  25B10  course  map  provided  for  the  present  research  reflected  partial 
differences  in  annex  titles  from  the  2005  program  of  instruction.  The  results  of  this  examination 
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suggest  that  the  resident  and  non-resident  instructors  interviewed  had  accurate  situation 
awareness  regarding  the  recency  and  overlap  of  their  course  materials. 


Table  3.  Content  (File)  Overlap  Among  Tobyhanna  and  Sacramento  HTRCs 


Annex 

% 

Overlap 

Explanation 

Overview  &  AIS  Security  (Both  HTRCs) 

63% 

(5/8) 

Sacramento  had  overview  documentation  and  IASO 

brief  that  Tobyhanna  did  not 

A  Plus  (Both  HTRCs) 

93% 

(54/58) 

Chapter  8  presentation  differed  across  locations; 

Tobyhanna  had  an  exam  that  Sacramento  did  not 

Network  Essentials  +  (Tobyhanna) 

Network  Essentials  (Sacramento) 

1 

26% 

(17/65) 

Fourteen  files  were  Chapter  documents  that 

Sacramento  had  but  Tobyhanna  did  not;  Another  14 
files  were  lecture  notes  that  Tobyhanna  had,  but 
Sacramento  did  not;  14  files  were  the  chapter 
presentation  slides  themselves,  which  were  different 
across  locations;  Sacramento  had  a  student  critique 
and  special  purpose  exercises  that  Tobyhanna  did  not 

TCP/IP  (Both  HTRCs) 

18% 

(3/17) 

Sacramento  had  practical  exercises  that  Tobyhanna 
did  not;  Sacramento  had  content  for  Lesson  2,  but 
Tobyhanna  did  not;  Lesson  plans  and  presentation 
slides  themselves  were  different 

Troubleshooting  (Both  HTRCs) 

100% 

(36/36) 

Routers  &  Switches  (Tobyhanna) 
Routers/Switches  (Sacramento) 

35% 

(8/23) 

Sacramento  had  Routers  folders  with  no  match  in 
Tobyhanna* s  Routers  &  Switches  annex;  Tobyhanna 
had  Router  folders  with  no  match  in  Sacramento’s 
Routers  annex 

Windows  XP/2003  Server  (Tobyhanna) 
Windows  XP/Outlook  (Sacramento) 
Windows  Server  2003  (Sacramento) 
Exchange  &  Outlook 
(Tobyhanna) 

Exchange  Server 
(Sacramento) 

49% 

(42/85) 

Sacramento  and  Tobyhanna  had  completely  non¬ 
overlapping  Windows  XP  content  and  completely 
non-overlapping  Outlook  content;  Tobyhanna  had 
content  for  installing  Exchange  2003  that  Sacramento 
did  not 

Systems  Administration/Network 
Management/Security  + 
(Tobyhanna)Security  +  (Sacramento) 

100% 

(21/21) 

Tobyhanna  did  not  have  systems  administration  or 
network  management  content  as  was  suggested  by  the 
annex  title 

UNIX  (Both  HTRCs) 

100% 

(23/23) 

SOLARIS  (Both  HTRCs) 

100% 

(18/18) 

MCS  (Tobyhanna) 

0% 

Sacramento  did  not  have  this  annex 

FBCB2  (Tobyhanna) 

0% 

Sacramento  did  not  have  this  annex 

TIMS  (Tobyhanna) 

0% 

Sacramento  did  not  have  this  annex 

DTOPS/AIS  (Tobyhanna) 

0% 

Sacramento  did  not  have  this  annex 

Communications  Exercise  (Tobyhanna) 

0% 

Sacramento  did  not  have  this  annex 

25B10  Field  Training  Exercise 
(Tobyhanna) 

0% 

Sacramento  did  not  have  this  annex 

Examination  of  the  25B10  course  materials  in  the  Tobyhanna  and  Sacramento  High-Tech 
Regional  Center  (HTRC)  sites  revealed  marked  differences  both  in  organization  and  content.  In 
both  HRTC  sites,  the  folder  structure  occasionally  required  students  to  drill  down  two  or  three 
folders  to  reach  a  single  document  or  presentation.  Sacramento  had  numerous  duplicate  folders 
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and  files.  Across  the  two  locations,  files  were  occasionally  named  differently,  but  upon 
examination  were  discovered  to  have  the  same  content.  Table  3  above  details  the  content 
differences  among  the  two  HTRCs  as  well  as  associated  explanations.  As  shown  in  the  table, 
content  overlap  in  the  majority  of  annexes  was  incomplete.  The  non-overlap  appeared  to  have 
been  due  largely  to  files  that  were  present  in  one  location  but  not  in  another. 

On  the  basis  of  this  analysis,  it  appears  that  the  LLC  staff  has  been  dutifully  posting 
25B 1 0  content  online,  but  that  uniform  access  to  MOSQ  content  has  not  yet  been  achieved.  To 
some  extent,  the  Fort  Gordon  LLC  may  not  be  able  to  facilitate  fully  uniform  content  due  to 
differences  in  programs  of  instruction  used.  Even  if  the  programs  of  instruction  were  the  same 
for  resident  and  non-resident  25B10  students,  there  would  exist  some  inconsistency  in  the 
learning  experience  across  both  types  of  learner.  For  instance,  the  head  instructor  from 
Sacramento  reported  that  his  HTRC  lacked  most  of  the  hands-on  training  equipment  necessary  to 
teach  25B10  (in  particular  the  annexes  on  the  Army  Battle  Command  System).  He  appeared  to 
be  unaware  that  some  simulations  to  support  his  instruction  were  available  in  the  eSignal 
downloads  portal.  A  process  involving  the  coordination  of  resident  instruction  managers  and  the 
LLC  staff  would  potentially  circumvent  the  challenges  to  presenting  uniform  course  content  by 
(1)  facilitating  communication  and  document  sharing  between  the  schoolhouse  and  lead 
instructors  in  the  Reserves;  (2)  simplifying  and  standardizing  the  interface  design  for  posting 
course  content;  and  (3)  developing  a  course  site  interface  that  provides  one-stop  access  to  lecture 
slides  and  related  simulations. 

Recommendations 

The  disconnect  between  LLC  outputs  and  organizational  outcomes  revealed  in  the 
present  analysis  should  not  lead  one  to  conclude  that  achieving  such  outcomes  for  MOSQ  is 
impossible.  Nor  should  this  finding  be  interpreted  that  the  Fort  Gordon  LLC  has  failed  to 
function  according  to  specification.  The  roles  of  the  LLC  staff  and  the  managers  of  resident 
instruction  evolved  independently  and  therefore  functioned  independently  at  the  time  this 
research  effort  was  conducted.  Closer  coordination,  guided  by  shared  understanding  of  how  the 
LLC  can  uniquely  improve  MOSQ  instruction,  would  increase  resident  instructors’  ability  to 
leverage  LLC  technologies  to  transform  the  classroom  and  enhance  proponent  outreach  through 
uniform  content  push  instead  of  individualized  content  pull.  Factors  outside  the  sphere  of 
influence  of  the  LLC  (i.e.,  Army-level  policies,  availability  of  public  computing  facilities,  and 
institutional  curriculum  review  processes)  must  be  addressed  at  a  higher  level,  but  data  collected 
at  the  lower  level  can  illustrate  the  need  and  direction  for  change. 

Assessment  of  Assignment-Oriented  Training 


The  Sample  -  25S10 

The  25S10  Course,  which  provides  MOSQ  instruction  for  “entry-level”  (Skill  Level  1) 
satellite  communications  operators-maintainers,  was  selected  as  an  example  for  studying  the  role 
that  the  Fort  Gordon  LLC  plays  in  enabling  the  cost-effective  delivery  of  assignment-oriented 
training  (AOT).  Regardless  of  the  curriculum-delivery  medium,  AOT  shortens  time  to 
competency  and  reduces  skill  decay  by  targeting  only  the  critical  tasks  that  students  will  need  to 
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perform  in  the  duty  position  they  assume  immediately  after  graduating.  Lengthy  Advanced 
Individual  Training  courses  (e.g.,  40+  weeks)  are  broken  down  into  coherent  technical  “tracks” 
that  target  the  skills  necessary  to  be  qualified  to  operate  unique  equipment  and/or  systems  in  a 
particular  echelon  unit  (Barrett,  2001;  Kinney,  2005).  For  example,  taken  as  a  whole,  the  25S10 
course  provides  common  core  instruction,  instruction  for  tactical  satellite  operation,  and 
instruction  for  strategic  satellite  operation.  All  25S10  students  take  the  common  core  and  then 
take  either  tactical  or  strategic  operator  instruction,  depending  on  the  type  of  unit  to  which  they 
expect  to  be  assigned.  Presumably  because  students  taking  AOT  spend  significantly  less  time  in 
the  schoolhouse,  course  throughput  may  increase  as  a  result,  further  reducing  the  cost  per  student 
to  deliver  instruction  (Wilson  &  Helms,  2003). 

LLCs  can  support  AOT  by  enabling  students  to  take,  as  needed,  additional  coursework 
via  distance  learning  if/when  their  duty  position  changes  (Vann,  2003).  The  alternate  track  of  the 
course  would  be  amenable  to  remote  delivery  and  self-study  because  students  already  have  a 
foundational  understanding  of  satellite  operation  on  which  to  build  (see,  for  example,  Shanley, 
Leonard,  &  Winkler,  2001).  Moreover,  allowing  students  to  take  the  alternate  track  of  the  course 
via  distance  learning  reduces  student  time  in  the  schoolhouse,  which  in  turn  reduces  training  cost 
and  enhances  readiness  because  learners  can  remain  with  their  unit  and  families  (Leonard, 
Winkler,  Hove,  et  al.,  2001).  The  25S10  course  was  selected  for  analysis  because  it  was  among 
the  first  of  the  MOSQ  courses  to  be  made  available  in  the  AOT  format  (at  that  time,  the  25S 
MOS  was  called  3 IS).  It  was  transitioned  to  AOT  format  in  2002. 

Assessment  Questions  &  Method 

The  analysis  of  AOT  impact  assessed  the  outcomes  of  delivering  the  25S10  course  in  an 
AOT  format.  Specifically,  the  following  questions  were  asked  and  associated  metrics  used  to 
focus  data  collection: 

o  Question:  Does  delivering  25S10  in  AOT  format  reduce  time  to  competency? 

o  Metric:  %  reduction  in  course  duration 
o  Question:  Does  delivering  25S 1 0  in  AOT  format  enhance  course  relevance? 

o  Metric:  Unit  commander/Training  supervisor  perceptions  of  course  relevance 
o  Metric:  Student  perceptions  of  course  relevance 
o  Question:  Does  delivering  25S10  in  AOT  format  increase  student  throughput? 

o  Metric:  %  increase  in  enrollment  and  graduation  after  LLC  implementation 
o  Question:  Does  LandWarNet  eSignal  enhance  the  delivery  of  AOT? 

o  Metric:  %  intended  target  audience  (i.e.,  25S10  follow-on  students)  registered  to 
use  the  system 

The  primary  means  used  to  capture  the  above  metrics  were  interviews.  The  Division 
Chief  for  25S  instruction  was  interviewed,  as  was  the  Chief  of  the  Resident  Training 
Management  Branch4,  the  LandWarNet  eSignal  Program  Manager,  the  Signal  Center  Director  of 
Training  who  oversaw  initial  AOT  implementation,  and  three  Signal  Center  Quality  Assurance 


4  During  the  course  of  this  research  effort,  this  person  assumed  a  new  position  as  the  Chief  of  Training  Management 
at  the  Signal  Regimental  NCO  Academy.  The  previous  position  is  referred  to  in  this  report  because  it  was  in  this 
capacity  that  the  interviewee  served  during  the  timeframe  of  interest. 
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personnel.  The  program  of  instruction  for  the  25S10  course  also  was  examined  to  partially 
determine  the  change  in  course  duration  affected  by  AOT.  Unfortunately,  data  from  the  Army 
Training  Requirements  and  Resources  System  (ATRRS)  database  could  not  be  used  directly  to 
answer  questions  about  student  throughput.  ATRRS-based  information  about  student  throughput 
was  retrieved  indirectly  through  discussion  with  the  Chief  of  Resident  Training  Management.  In 
addition,  because  the  available  ATRRS  records  did  not  go  back  far  enough  to  compare  course 
enrollment/graduation  numbers  before  and  after  the  implementation  of  AOT,  discussion  centered 
on  enrollment/graduation  trends  over  the  years  of  offering  25S10  in  AOT  format.  Archival 
studies  conducted  by  the  Quality  Assurance  Office  and  by  the  TRADOC  Analysis  Center 
(Kinney,  2005)  were  examined  for  information  about  the  impact  of  AOT  on  Soldiers  and  units  in 
the  field. 

Summary  of  Findings 

The  results  of  the  25S10  assessment  tentatively  suggest  that  providing  Advanced 
Individual  Training  courses  in  AOT  format  can  significantly  reduce  course  duration,  thus  saving 
money  and  potentially  reducing  time  to  competency.  Strong  statements  about  the  enhancement 
of  operational  performance  or  unit  readiness  due  to  reduced  time  to  competency  could  not  be 
made,  but  no  negative  impact  on  performance  was  demonstrated.  Training  and  human  resources 
management  issues  appeared  to  prevent  AOT  from  achieving  its  full,  envisioned  impact  on 
student  throughput  and  online  course  delivery  through  the  Fort  Gordon  LLC. 

Outreach  to  operational  units  who  received  AOT  graduates  appeared  to  be  quite 
challenging.  Despite  a  smoothly  functioning  process,  surveys  administered  remotely  by 
proponent  quality  assurance  personnel  received  few  responses.  In  contrast,  an  archival  study  of 
AOT  effectiveness  (Kinney,  2005)  produced  a  wealth  of  data,  but  it  lasted  two  and  a  half  years 
and  involved  six  site  visits  conducted  by  a  team  of  researchers.  These  findings  suggest  that  the 
success  of  future  assessments  of  LLC  impact  on  unit  outcomes  will  depend  on  the  coordinated 
effort  of  parties  at  the  proponent  and  Army  level,  including  the  TRADOC  Analysis  Center. 

Detailed  Findings 

Time  to  Competency.  Using  the  reduction  in  course  duration  as  a  metric,  delivering  the 
25S10  course  in  AOT  format  significantly  reduced  time  to  competency.  Prior  to  AOT,  the  25S10 
course  lasted  approximately  46  weeks5.  According  to  the  2007  Fiscal  Year,  2nd  Quarter  program 
of  instruction,  students  taking  25S10  in  the  AOT  format  only  spent  26  weeks  in  the  schoolhouse. 
This  represents  a  reduction  of  43%  time  in  the  schoolhouse.  An  analogous  reduction  in  cost 
associated  with  housing  each  student  could  be  expected,  although  travel  costs  to  and  from  the 
schoolhouse  would  not  be  affected  (and  could  possibly  increase  if  re-assigned  graduates  returned 
to  the  schoolhouse  for  alternate-track  instruction). 

Course  Relevance.  Past  assessments  of  AOT  conducted  by  the  Signal  Center  have 
involved  capturing  commander/supervisor  and  student  impressions  of  course  relevance  via 


5  Because  a  pre-AOT  version  of  the  25S10  course  was  not  available,  the  duration  of  the  25S10  course  prior  to 
implementation  of  AOT  format  is  the  average  of  two  estimates  (42  weeks  and  50  weeks)  given  by  two  different 
interviewees. 
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survey.  Surveys  were  administered  by  the  Quality  Assurance  Office  via  the  Automatic  Survey 
Generation  (AUTOGEN)  tool.  The  most  recent  AUTOGEN  survey  on  25S10  was  administered 
from  April  to  Nov  2005,  and  data  were  reported  internally  in  July  2006.  This  survey  asked  25S 1 0 
graduates  (tactical  and  strategic  tracks)  and  their  supervisors  to  report  on  their  ability  to  perform 
critical  technical  tasks  to  standard6.  In  the  present  research,  the  intent  was  to  work  with  this 
existing  survey  data  such  that  duplication  of  effort  could  be  avoided  and  outreach  beyond  the 
institutional  setting  could  be  achieved.  Unfortunately,  the  response  rate  (for  both  graduates  and 
supervisors)  to  the  surveys  was  too  low  to  draw  conclusions.  The  response  rate  for  supervisors  of 
tactical  25S10  graduates  was  0%.  For  graduates  it  was  3%.  For  the  strategic  track  25S10 
graduates  and  supervisors,  the  response  rate  was  0%. 

In  the  absence  of  AUTOGEN  survey  data,  a  preliminary  assessment  of  AOT  conducted 
by  the  TRADOC  Analysis  Center  (Kinney,  2005)  offered  a  partial  answer  to  the  question  of 
AOT  impact  on  unit  outcomes.  In  this  study,  research  team  members  visited  1 7  signal  battalions 
in  six  locations,  conducting  surveys  and  interviews  with  the  graduates  of  four  AOT  courses 
(25F10,  25P10,  25Q10,  and  25Q10)  and  their  supervisors.  According  to  Kinney,  60%  (9/15)  of 
25S 10  graduate  supervisors  reported  that  AOT-trained  Soldiers  were  “basically  the  same”  as 
traditionally  trained  Soldiers.  Twenty-seven  percent  (4/15)  claimed  that  AOT-trained  Soldiers 
were  worse,  and  13%  (2/15)  claimed  they  were  better.  In  any  case,  supervisors  were  unanimous 
in  reporting  that  they  treated  AOT-trained  and  traditionally  trained  Soldiers  the  same  with  regard 
to  unit  training  and  work  assignments.  These  results  tentatively  suggest  that  the  readiness  impact 
of  AOT  may  be  seen  primarily  in  greater  institutional  training  efficiency,  rather  than  enhanced 
unit  proficiency. 

Another  way  to  look  at  course  relevance  is  to  determine  the  percentage  of  students  who 
are  placed  in  units  for  which  they  have  received  the  “correct”  track  of  the  course.  As  noted  in 
Shanley,  Crowley,  Lewis,  Masi,  Straus,  Leuschner,  et  al.  (2005)  and  Kinney  (2005),  AOT  places 
a  nontrivial  burden  on  personnel  management  systems  to  link  individuals’  training  and  duty 
assignments.  Kinney  (2005)  detailed  several  human  resources  management  issues  that  gave  rise 
to  the  malassignment  of  AOT  graduates,  including  (1)  lack  of  automated  databases  for  tracking 
and  assigning  Soldiers  with  unconventional  identifiers  (i.e.,  additional  skill  identifiers  instead  of 
MOS  designations);  (2)  confusion  over  manual  data  entry  responsibilities;  and  (3)  confusion  over 
how  to  assign  Soldiers  with  unconventional  identifiers.  Interviews  with  both  the  Division  Chief 
and  the  Chief  of  the  Resident  Training  Management  Branch  indicated  that  the  accuracy  of 
student  placement  was  very  high  (>95%).  Kinney  (2005)  largely  supports  these  observations, 
citing  an  8%  malassignment  rate  for  AOT  overall  and  a  10%  malassignment  rate  for  25S10. 

Student  Throughput.  As  described  previously,  delivering  the  25S10  course  in  an  AOT 
format  reduced  the  length  of  the  course  by  43%.  Reducing  the  course  by  this  much  time 
introduced  the  possibility  that  the  course  may  be  delivered  roughly  twice  as  often,  thus 
approximately  doubling  the  student  load  each  academic  year.  Increased  throughput  has  obvious 
implications  for  reducing  the  cost  per  student  (cost  of  administering  the  course  per  student  would 
be  roughly  halved)  and  enhancing  readiness  (enhanced  training  and  personnel  status),  which 
were  expected  benefits  of  AOT  detailed  by  the  initial  planners  of  the  Fort  Gordon  LLC  (Wilson 
&  Helms,  2003).  Interviews  with  both  the  25S  Division  Chief  and  the  Chief  of  the  Resident 


6  Consistent  with  TRADOC  practice,  “standard”  was  defined  in  the  survey  as  80%  GO. 
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Training  Management  Branch  indicated,  however,  that  student  throughput  has  not  changed  since 
the  implementation  of  AOT.  The  Resident  Training  Management  Chief  citied  a  level  trend  in 
ATRRS  enrollment/graduation  data  since  the  initial  years  of  AOT. 

The  25S  Division  Chief  provided  some  explanation  for  this  level  trend.  He  reported  that 
the  introduction  of  AOT  has  led  instructors  to  become  more  specialized  such  that  their  ability  to 
teach  across  topics  has  become  more  limited.  Instead  of  the  same  instructor  teaching  both  the 
tactical  and  strategic  tracks  of  the  course,  different  instructors  teach  each  segment  because  they 
are  held  at  the  same  time  (because  both  tactical  and  strategic  students  begin  at  the  same  time  by 
taking  the  common  core).  Teaching  the  course  in  overlapping  tracks  therefore  would  require 
more  instructors,  rather  than  the  same  number,  and  more  instructors  are  not  available.  25S 
historically  has  been  a  MOS  with  persistent  shortages,  as  reflected  in  the  monthly  readiness 
reports  of  the  Chief  of  Staff  of  the  Army  (Shanley,  Leonard,  &  Winkler,  2001).  Soldiers  with  the 
necessary  skills  to  operate  satellite  communications  equipment  therefore  are  assigned  to  units 
that  need  them. 

Online  Delivery  of  AOT.  The  adoption  of  the  Fort  Gordon  LLC  to  deliver  AOT  is 
reflected  in  the  percentage  of  the  intended  target  audience  registered  to  use  the  system  for  this 
purpose.  In  the  case  of  the  25S 10  course,  the  intended  target  audience  would  be  those  students 
who  are  reassigned  after  graduation  and  must  complete  a  different  track  of  25S10  instruction. 
According  to  the  initial  plans  for  AOT,  the  Gordon  LLC  technologies  would  be  used  to  deliver 
this  follow-on  instruction  (Wilson  &  Helms,  2003).  Currently  however,  the  LLC  is  not  used  to 
deliver  follow-on  25S10  instruction,  largely  due  to  difficulties  with  the  automated  management 
of  individual  training  assignments.  Students  instead  return  to  the  schoolhouse  to  complete  their 
training. 

That  said,  true  cost  savings  associated  with  delivering  follow-on  AOT  training  via  the 
LLC  are  demonstrated  when  the  intended  user  audience  is  very  large.  In  the  case  of  25S10,  this 
is  not  the  case.  The  25S  Division  Chief  estimated  that  only  a  very  small  percentage  (fewer  than 
5%)  of  25S10  graduates  require  follow-on  instruction.  The  cost  savings  associated  with  reducing 
course  duration  therefore  reflect  the  majority  of  cost  savings  that  could  be  achieved  via  AOT,  at 
least  for  the  25S 10  course. 

Recommendations 

Recommendations  for  enhancing  the  impact  of  AOT  on  unit  outcomes  must  be  directed 
outside  of  the  scope  of  the  LLC.  Factors  affecting  impact  apparently  were  driven  by  a  lack  of 
policy  on  how  to  interpret  and  manage  unconventional  identifiers  (i.e.,  additional  skill 
identifiers)  for  tracking  training  status  and  making  unit  assignments.  If  developed,  this  policy 
could  be  used  to  design  databases  for  automatically  performing  these  management  functions. 
Perhaps  more  importantly,  such  policy  could  serve  as  a  springboard  for  developing  personnel 
management  practices  that  align  the  particular  capabilities  of  individual  Soldiers  to  the  specific 
competency  requirements  of  units.  Such  practices  would  significantly  expand  the  possibilities  for 
implementing  lifelong  learning  on  an  Army-wide  scale. 
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Assessment  of  Simulations 


The  Sample  -  25N10  &  Joint  Network  Node  (JNN)  Simulation 

The  25N10  course  provides  qualification  instruction  for  the  25N  MOS  created  in 
October,  2005.  It  prepares  “entry-level”  (Skill  Level  1)  nodal  network  systems  operators- 
maintainers  to  perform  basic  operations  with  the  Joint  Network  Node  (JNN)  Network,  which  the 
Army  has  selected  to  replace  legacy  Mobile  Subscriber  Equipment.  At  the  time  of  writing  this 
report,  the  25N 1 0  course  did  not  have  a  non-resident  version  and  course  content  was  not  hosted 
in  the  Fort  Gordon  LLC  for  resident  students.  All  instruction  was  performed  in  the  classroom  via 
lecture  and  practical  exercise.  Since  its  inception,  the  25N10  course  has  made  use  of  equipment 
simulations  to  facilitate  classroom  instruction  and  practical  exercise  performance.  This  practice 
is  consistent  with  the  historical  use  of  training  technology  (i.e.,  interactive  videodisc)  to  support 
Army  communications  training  in  the  schoolhouse  (Winkler  &  Polich,  1990). 

Specifically,  simulated  JNN  equipment  was  used  to  supplement  hands-on  training  by 
giving  25N 10  students  the  opportunity  to  practice  basic  operations  on  a  desktop  computer  while 
waiting  to  conduct  hands-on  training  with  actual  equipment.  As  described  earlier  in  this  report, 
equipment  simulations  such  as  the  JNN  courseware  use  the  scaffolding  approach  to  instruction, 
walking  students  through  an  Acquire  Practice  Validate  sequence  for  each  skill  to  be 
learned.  The  same  simulation  used  in  the  25N10  course  also  has  been  made  available  as  a  stand¬ 
alone  download  in  the  Fort  Gordon  LLC.  Learners  outside  of  the  schoolhouse  setting  (e.g., 
deployed  signal  Soldiers)  who  have  common  access  cards  can  retrieve  the  simulation  from 
LandWarNet  eSignal,  and  many  have.  As  of  September  17,  2007  the  JNN  simulation  had  been 
downloaded  2,371  times  for  refresher  training  or  preparation  for  hands-on,  on-the-job  learning 
with  new  equipment. 

By  studying  the  use  of  equipment  simulations  in  schoolhouse  instruction  and  by  non¬ 
resident  learners,  it  is  possible  to  assess  their  effectiveness  as  learning  tools  in  a  variety  of 
settings.  It  is  because  of  JNN  simulation  availability  in  multiple  learning  contexts  that  it  and  the 
25N10  course  were  selected  for  analysis.  Simulations  have  been  proposed  to  be  at  least  a  partial 
alternative  to  hands-on  training,  thereby  reducing  time  on  equipment  and  equipment-related 
costs,  while  simultaneously  maintaining  pedagogical  standards  (e.g.,  Wilson  &  Helms,  2003; 
Winkler  &  Polich,  1990).  It  should  be  noted  that  one  annex  of  only  one  course,  using  a  single 
suite  of  simulations  provided  by  a  single  contractor,  was  assessed  in  the  present  research.  In 
addition,  the  25N10  course,  having  never  been  administered  without  simulations,  cannot  permit 
before-after  comparisons  of  some  metrics  of  interest,  including  time  savings,  equipment 
requirements,  and  student  throughput7.  A  rigorous,  scientific  study  has  been  conducted  to  assess 
the  use  of  simulations  for  Army  communications  training  (Winkler  &  Polich,  1990),  which  was 
used  to  provide  a  context  for  interpreting  the  findings  of  the  present  research.  On  the  basis  of  this 
additional  context,  some  specification  of  the  generalizability  of  the  present  findings  is  provided 
below. 


7  Other  types  of  analyses  to  capture  these  metrics  were  performed  as  part  of  this  research.  Consistent  with  the 
qualitative  causal  approach,  these  alternative  analyses  focused  on  the  processes  that  would  enable  such  outcomes  as 
time  savings,  equipment  requirements,  and  throughput. 


29 


Assessment  Questions  &  Method 

The  analysis  of  simulation  implementation  and  use  for  lifelong  learning  focused  on  the 
output  and  outcomes  associated  with  using  equipment  simulation  to  provide  schoolhouse  and  on- 
the-job  instruction  on  basic  Joint  Network  Node-Network  operations.  Specifically,  the  following 
questions  were  asked  and  associated  metrics  used  to  focus  data  collection: 

o  Question:  Does  the  use  of  simulations  in  25N 1 0  save  time  on  equipment? 

o  Metric:  %  reduction  in  time  on  equipment 
o  Question:  Does  the  use  of  simulations  in  25N10  enhance  instructional  efficiency? 

o  Metric:  %  instructors  who  report  using  increased  class  time  to  enhance  instruction 
o  Question:  Does  posting  the  JNN  simulation  on  the  LLC  enable  uniform  access  to  JNN- 
related  learning  content? 

o  Metric:  Equivalence  of  learning  experience  across  resident  and  non-resident 
learners 

o  Question:  Does  the  JNN  simulation  enable  just-in-time  competency? 

o  Metric:  %  on-the-job  simulation  users  who  report  that  the  JNN  simulation  enabled 
just-in-time  competency 

o  Question:  Does  the  use  of  simulations  for  25N 1 0  broaden  thinking  about  where  and  how 
technical  training  can  be  accomplished? 

o  Metric:  %  students  and  instructors  who  believe  that  simulations  can  serve  as  a 
substitute  for  hands-on,  classroom  training 

To  capture  metric  data,  a  combination  of  focus  groups,  classroom  observation,  and 
survey  was  used.  Specifically,  two  focus  groups  were  conducted,  one  with  25N 10  students  ( N= 
16)  and  one  with  25N10  instructors  (N=  5).  The  purpose  of  these  focus  groups  was  to  gather 
impressions  of  the  effectiveness  and  utility  of  the  JNN  simulation  for  resident  25N10  instruction 
in  general.  Classroom  observations  (with  different  students  and  instructors  than  those 
interviewed  initially)  then  were  conducted  during  one  of  the  annexes  of  the  25N 10  program  of 
instruction.  This  annex  focused  on  the  basic  functions  of  the  Promina  system.  Two  lecture 
simulation  ->  hands-on  training  sequences  (Database  Management  and  Basic  Trunks  Services) 
from  this  annex  were  observed.  It  .should  be  noted  that  this  small  sample  of  students  and  the 
particular  course  annex  selected  may  not  be  representative,  however  the  findings  from  the 
present  analysis  is  consistent  with  previous  scientific  analyses  and  with  adult  learning  theory  so 
the  sample  is  likely  not  an  outlier.  An  online  survey  of  on-the-job  users  of  the  JNN  simulation 
was  administered  via  LandWarNet  eSignal.  The  survey  was  posted  in  the  Downloads  site  of 
eSignal  and  was  open  for  approximately  3  weeks,  but  received  no  responses. 

Summary  of  Findings 

The  JNN  simulation  was  actively  used  in  the  25N 1 0  course  and  frequently  downloaded 
from  the  eSignal  Downloads  site.  Instructors  used  the  time  saved  by  learning  via  desktop 
simulation  to  enhance  the  training  they  provided  with  hands-on  equipment.  Use  of  equipment 
simulations  allowed  for  greater  practice  time  on  equipment  procedures,  which  has  positive 
theoretical  implications  for  skill  retention.  Instructors  and  students  alike  endorsed  simulation- 
supported  instruction  and  had  forward-thinking  views  about  how  to  use  simulations  to  improve 
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learning.  Anecdotal  evidence  indicated  that  the  stand-alone  JNN  simulation  download  could  be 
used  in  a  manner  similar  to  schoolhouse  classroom  to  enable  just-in-time  competency 
development.  However,  more  research  will  be  necessary  to  understand  how  simulation 
downloads  are  used  by  distributed  learners.  The  impact  of  the  JNN  simulation  on  just-in-time 
competency  development  could  not  be  determined  due  to  lack  of  survey  response  data. 

The  present  findings,  coupled  with  previous  scientific  study  (Winkler  &  Polich,  1990) 
suggested  that  desktop  simulations  can  partially  substitute  for  hands-on  training.  The  general 
caution  that  simulations  cannot  replace  hands-on  instruction  was  supported  by  the  25N10 
classroom  observations.  Inconsistencies  between  the  JNN  simulation  and  the  actual  equipment 
caused  difficulty  for  some  students  as  they  transferred  automatic,  but  incorrect  skills  from  their 
desktop  computer  to  the  hands-on  training  equipment.  Characteristics  of  the  actual  JNN 
equipment  omitted  from  the  simulations  had  to  be  addressed  using  hands-on  training  time. 
Instructors  observed  that  a  faster  contracting  cycle  would  significantly  accelerate  the  delivery  of 
up-to-date  simulations. 

Detailed  Findings 

Time  on  equipment.  Focus  group  interviews  revealed  a  difference  of  opinion  regarding 
the  degree  to  which  simulations  reduced  the  time  required  to  achieve  proficiency  with  actual 
JNN  equipment.  25N 10  students  reported  that  little  transferable  learning  was  accomplished  with 
the  simulations,  citing  fidelity  shortfalls  and  simplistic  learning  objectives  as  reasons  for  limited 
utility.  For  example,  students  indicated  that  they  could  get  the  correct  answers  during  the 
Practice  and  Validate  phases  of  the  simulation  without  knowledge  of  the  underlying  principles  of 
how  the  simulated  equipment  worked.  They  also  stated  that  the  simulations  did  not  allow  them  to 
make  mistakes  that  they  could  make  on  the  actual  equipment,  thus  preventing  them  from 
learning  how  to  undo  their  actions  in  an  operational  situation. 

In  contrast,  25N10  instructors  reported  that  using  simulations  to  practice  foundational 
skills  allowed  students  to  achieve  proficiency  with  actual  equipment  faster  than  they  would 
without  such  practice.  Although  they  acknowledged  limitations  of  simulations,  instructors  saw 
accelerated  learning  as  a  key  benefit  of  their  adoption.  Instructors  also  noted,  however,  that 
enhanced  proficiency  with  hands-on  equipment  did  not  actually  reduce  student  time  on  the 
equipment,  but  rather  allowed  instructors  to  make  better  use  of  the  equipment  time  available. 

Classroom  observation  supported  both  of  these  contrasting  perspectives.  First,  the 
observations  that  students  made  about  the  simulations  were  partially  validated.  The  simulations 
did  have  some  functional  dissimilarity  from  the  actual  equipment,  which  affected  skill  transfer 
for  some  students.  A  small  minority  of  students  found  it  frustrating  to  override  incorrect,  but 
automatic  skills  developed  using  the  simulations  (e.g.,  use  of  arrow  keys  and  command  line  text) 
when  they  transitioned  to  hands-on  training.  Unlearning  automatic  skills  increased  hands-on 
equipment  time  for  these  students  relative  to  those  who  did  not  have  such  difficulty.  Skill 
development  using  the  desktop  simulation  was  subject  to  decay,  and  when  students  made 
mistakes  on  basic  procedures,  they  did  not  know  how  to  undo  them  on  the  actual  equipment.  In 
addition,  there  were  aspects  of  the  actual  equipment,  such  as  status  lights,  which  convey 
important  information  about  the  success  of  task  execution,  that  were  not  part  of  the  simulations. 
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Students’  questions  during  the  hands-on  training  that  were  observed  often  addressed  the  meaning 
of  the  status  lights  and  their  relation  to  the  task  at  hand. 

Second,  classroom  observations  validated  the  instructors’  report  of  reduced  time  to 
competency  and  enhanced  use  of  class  time.  Although  printed  materials  with  detailed 
instructions  were  available  to  students  during  hands-on  instruction,  at  least  half  of  the  students 
did  not  need  to  refer  to  them  to  complete  hands-on  exercises.  The  amount  of  time  spent  acquiring 
the  analogous  skills  using  the  desktop  simulation  was  approximately  10-15  minutes.  Students 
using  hands-on  equipment  moved  quickly  through  the  basic  skills,  and  instructors  optimized 
(rather  than  shortened)  the  learning  experience  by  drawing  linkages  between  task  performance 
and  the  components  of  the  equipment  not  reflected  in  the  simulation  (e.g.,  status  lights  and 
installation  activities).  Given  the  roughly  equivalent  skill  transfer  associated  with  simulation- 
supported  instruction  versus  hands-on  only  training  found  in  Winkler  and  Polich  (1990),  one 
might  assume  that  the  same  amount  of  time  spent  learning  on  the  desktop  simulations  would 
have  been  spent  learning  on  the  hands-on  training  equipment.  This  represents  a  significant  time 
savings  that  could  be  used  to  enhance  the  hands-on  training  experience,  even  though  time  on 
equipment  was  not  reduced. 

An  important  implication  of  coupling  desktop  simulation  with  hands-on  training  was  that, 
in  one  sense,  time  on  equipment  actually  increased.  That  is,  students  had  the  opportunity  to 
increase  the  time  spent  on  practicing  equipment  procedures,  regardless  of  whether  the  equipment 
was  real  or  virtual.  The  Acquire  4  Practice  Validate  sequence  used  in  the  desktop  simulation 
required  students  to  practice  the  same  procedure  three  times,  whereas  the  hands-on  training 
would  only  have  supported  going  through  the  procedure  enough  times  to  demonstrate 
proficiency.  Moreover,  students  could  practice  the  procedures  on  the  desktop  simulations  as 
many  times  as  they  wished,  depending  on  their  self-estimate  of  proficiency.  The  additional 
training  time,  which  could  be  as  short  as  5  minutes  (i.e.,  the  observed  average  time  spent  on 
hands-on  training  demonstrating  proficiency)  or  as  long  as  students  wished,  would  enhance  skill 
retention  for  which  initial  levels  of  proficiency  is  the  leading  predictor  (Arthur,  Bennett,  Stanush, 
&  McNelly,  1998).  The  present  findings  are  consistent  with  Winkler  and  Polich  (1990). 

Instructional  Efficiency.  Instructor  interviews  in  both  focus  groups  and  classroom  settings 
indicated  that,  in  general,  time  on  equipment  was  not  reduced  by  simulations,  but  rather  that 
simulations  enhanced  scheduled  hands-on  training.  25N10  classroom  observation  supported  this 
perspective  by  revealing  that  hands-on  training  was  used  to  increase  the  basic  proficiency 
accomplished  via  desktop  simulation  practice  and  to  develop  skills  not  addressed  by  the 
equipment  simulations.  As  described  in  the  previous  section,  instructors  enhanced  hands-on 
training  by  explaining  the  meaning  of  equipment  status  lights,  providing  students  with 
opportunities  to  make  and  undo  errors,  and  highlighting  links  between  the  classroom  theory  and 
practical  procedures. 

Equivalence  of  learning  experience.  To  the  extent  that  technical  skill  development  using 
simulations  involves  analogous  instructional  strategies  across  diverse  learning  contexts  (e.g., 
classroom  vs.  field  settings),  the  learning  experience  and  projected  effectiveness  of  the  training 
may  be  considered  to  be  equivalent.  Clark  (1994)  has  argued  (and  others,  e.g.,  Sitzmann, 

Kraiger,  Stewart,  &  Wisher,  2006,  have  demonstrated)  that  instructional  technology  is  only  as 
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effective  as  the  pedagogy  it  supports.  The  training  and  education  literature  indicates  that  some 
key  components  of  effective  pedagogy,  especially  as  it  relates  to  technical  skill  development  and 
computer-based  instruction  include  scaffolding,  interactivity,  and  application  (Abell,  2003; 
Firdyiwek,  1999;  Hays,  Stout,  and  Ryan-Jones,  2005).  Particularly  for  those  people  using 
simulations  to  learn  outside  of  the  classroom  context,  therefore,  structured  engagement  with 
content  and  contact  with  others,  i.e.,  peers  and  mentors,  during  the  learning  process  should  be 
considered  critical  to  learning  outcomes.  This  is  especially  true  when  simulations  deviate  from 
actual  equipment  function,  which  is  commonly  the  case  because  equipment  versions  change 
faster  than  their  associated  desktop  simulations  do. 

In  the  25N10  course,  simulations  were  used  as  learning  tools  within  the  context  of 
lectures  about  how  the  system  works  generally.  The  lecture  slides  presented  high-level  bullet 
points  and  diagrams,  and  the  instructor  augmented  this  content  with  additional  information  about 
how  the  equipment  functions  and  its  integration  into  the  larger  signal  environment.  The 
instructor  also  engaged  the  students  by  asking  questions  and  using  humor.  Classroom 
observations  further  indicated  that  25N10  students  and  instructors  had  high  levels  of  interaction 
while  using  the  simulations  and  transitioning  from  simulations  to  hands-on  equipment.  In 
particular,  instructors  made  students  aware  of  the  specific  ways  the  simulations  behaved 
differently  from  the  actual  equipment.  For  example,  during  the  classes  observed,  there  was  a 
particular  function  for  which  the  simulation  behaved  in  exactly  the  opposite  way  than  the  actual 
equipment.  For  the  present  researcher,  this  difference  caused  a  great  deal  of  confusion  while 
observing  hands-on  training,  but  students  had  been  made  aware  of  it  in  an  earlier  lecture  (not 
observed)  and  so  knew  to  work  around  the  problem.  Instructors  also  encouraged  students  to  use 
the  simulations  to  enhance  retention  through  repeated  practice  before  and  after  hands-on  training. 
All  of  these  practices,  according  to  commonly  accepted  learning  theory  (e.g.,  American  Distance 
Education  Consortium,  2003;  Chickering  &  Gamson,  1987;  University  of  Illinois,  Urbana- 
Champaign,  1 999),  should  be  expected  to  maximize  the  learning  benefit  of  using  simulations. 

Unfortunately,  due  to  the  lack  of  response  to  the  JNN  survey  administered  to  eSignal 
users,  a  direct  comparison  could  not  be  made  between  the  simulation-supported  learning 
experience  in  the  classroom  and  in  the  field.  Some  differences  in  learning  experience  may  be 
assumed,  however,  given  that  the  JNN  simulation  download  was  not  coupled  with  25N  course 
materials,  such  as  lecture  slides.  Possibly  more  importantly,  the  download  was  not  accompanied 
by  explanations  of  the  differences  between  the  desktop  simulation  and  various  versions  of  the 
actual  JNN  equipment.  Discussions  in  the  eSignal  technical  forums  would  be  one  place  to 
provide  peer  mentorship  on  these  topics,  but  such  discussions  were  not  observed  in  the  analyses 
of  the  forums  detailed  later  in  this  report.  Discussions  did  address  difficulties  in  downloading  the 
large  files  associated  with  the  JNN  simulation  and  brought  up  questions  of  versioning. 

An  alternative  source  of  mentorship  for  on-the-job  learners  might  include  more  senior 
Soldiers  who  have  greater  experience  with  JNN  operations.  This  possibility  was  presented  as 
likely  in  the  focus  groups  with  25N10  students.  One  unit  trainer  interviewed  as  part  of  this 
research  effort  reported  that  he  used  the  JNN  simulation  download  to  teach  12  subordinates  using 
a  classroom  setting.  The  classroom  setting  enabled  students  to  learn  individually  at  their  own 
workstations  but  in  a  group  setting  similar  to  that  observed  for  25N10  instruction.  Beyond  these 
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indirect  inferences  and  anecdotal  evidence,  no  further  conclusion  may  be  made  about  the 
equivalence  of  the  learning  experience  for  schoolhouse  and  on-the-job  learners. 

Just-in-time  competency.  Unfortunately,  due  to  lack  of  response  to  the  JNN  survey,  the 
ability  of  desktop  simulation  to  enable  just-in-time  competency  could  not  be  analyzed. 

Download  data  alone  cannot  be  used  as  a  proxy  for  just-in-time  competency  because  it  is 
unknown  what  is  done  with  the  simulations  after  they  have  been  retrieved.  The  previous  example 
of  the  unit  trainer  who  conducted  classroom  JNN  training  with  the  simulation  download  suggests 
that  just-in-time  competency  can  be  supported  by  simulations,  however.  This  person  actively 
sought  out  the  simulation  to  conduct  training  while  his  unit  waited  to  receive  the  actual 
equipment  assigned  to  them.  Greater  detail  on  how  the  downloads  are  used  will  go  a  long  way 
toward  understanding  LLC  impact  on  just-in-time  competency. 

Broader  thinking.  As  described  previously,  it  has  been  proposed  that  simulations  may  be 
considered  at  least  a  partial  substitute  for  hands-on  training  with  actual  equipment  (Wilson  & 
Helms,  2003;  Winkler  &  Polich,  1990).  Such  a  substitution  would  potentially  reduce  the  cost- 
per-student  of  MOSQ  instruction  in  the  schoolhouse  (and  at  other  locations)  by  reducing  the 
need  for  costly  training  equipment.  In  addition,  the  education  of  non-resident  learners,  such  as 
those  in  the  Reserves  who  complete  their  instruction  at  High-Tech  Regional  Centers,  could  be 
made  more  equivalent  to  that  of  students  in  the  schoolhouse  by  increasing  access  to  equipment 
function.  The  adoption  of  simulations  as  an  alternative  to  hands-on  training  requires  broader 
thinking  about  what  may  be  considered  a  learning  environment  such  that  learners  ( 1 )  seek  and 
use  simulations  rather  than  awaiting  classroom  instruction;  (2)  actively  use  simulations  to  aid  in 
initial  learning  and  retention;  and  (3)  encourage  the  use  of  simulations  for  the  training  of  others. 

Members  of  both  the  25N 10  student  and  instructor  focus  groups  unanimously  supported 
the  use  of  equipment  simulations  for  technical  skill  development.  Moreover,  they  had  forward- 
thinking  ideas  about  simulation  properties  that  would  better  address  higher-level  cognitive  skills. 
These  ideas  included  embedding  short  scenarios  into  the  simulation  instruction  that  required 
learners  to  troubleshoot  equipment  malfunctions  and  networking  simulated  systems  together 
such  that  students  could  experience  the  challenges  to  achieving  the  interoperability  of  joint 
network  nodes.  That  said,  there  also  was  unanimous  agreement  among  the  members  of  both  the 
student  and  instructor  focus  groups  that  simulations  should  not  be  considered  a  substitute  for 
hands-on  training  with  actual  equipment.  Their  concerns  stemmed  from  the  conclusion  that 
simulations  could  not  fully  replicate  the  functionally  of  actual  equipment.  These  conclusions 
were  supported  by  classroom  observations,  which  indicated  that  dissimilarity  partially 
challenged  the  learning  process.  Winkler  and  Polich  (1990)  also  found  that  compared  to  students 
trained  solely  on  hands-on  equipment,  students  trained  primarily  using  alternative  technology 
(interactive  videodisc)  had  slightly  higher  error  rates  in  hands-on  test  conditions. 

Achieving  the  technological  sophistication  necessary  to  fully  replicate  JNN  functionality 
would  be  costly,  technically  difficult,  and  would  require  that  the  contracting  cycle  (to 
develop/modify  simulations)  keep  pace  with  the  equipment  update  cycle.  It  is  unknown  whether 
effort  and  resources  invested  in  developing  high-fidelity  networked  simulations  would  produce 
the  cost-benefit  tradeoff  expected  from  simulations. 
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Recommendations 


More  information  on  the  impact  of  simulation  downloads  on  unit  outcomes  is  required. 
LLC  staff  could  support  efforts  to  collect  external  data  on  simulation  use  in  the  field  by 
providing  relatively  simplistic  methods  for  reporting  on  effectiveness.  For  instance,  a  rating 
capability  and  comments  area  associated  with  each  download  in  the  Downloads  page  of  eSignal 
would  present  a  quick  and  simple  way  for  remote  users  to  provide  feedback  and  information 
about  simulation  functionality.  Follow-on  discussion  led  by  the  eSignal  technical  forum 
facilitators  could  provide  further  detail  on  how  simulations  are  used  in  the  field.  This  information 
would  be  useful  not  only  for  assessing  simulation  use  but  also  for  communicating  to  the  field 
Army  the  training  caveats  that  must  be  observed  in  order  to  use  the  simulations  effectively.  In 
this  way,  the  proponent  could  assist  in  the  mentoring  process  for  distributed  learners  through  the 
LLC  staff.  Closer  coordination  between  the  LLC  staff  and  the  resident  instruction  managers 
would  increase  the  likelihood  that  other  supporting  materials,  such  as  lecture  slides,  were  posted 
along  with  simulation  downloads.  Some  efforts  to  provide  this  type  of  supporting  information 
were  begun  during  the  course  of  this  research  effort. 

Taking  a  longer-term  view,  for  simulations  to  have  an  impact  on  the  bottom  line  (i.e., 
cost  savings  associated  with  reduced  time  on  equipment),  as  well  as  retention  (Arthur  et  al., 

1998)  they  will  have  to  better  replicate  system  functionality  through  a  combination  of  in-depth 
task  analysis,  effective  instructional  design,  and  an  accelerated  contracting  cycle.  An  in-depth 
task  analysis  would  enable  simulation  developers  to  better  understand  both  the  explicit  and 
implicit  tasks  that  must  be  executed  to  operate  equipment.  Instructional  design  then  could 
exercise  more  of  the  higher-level  cognitive  skills  that  trainees  must  use  to  monitor,  troubleshoot, 
and  integrate  the  equipment  through  a  combination  of  scaffolding  and  scenario-based  training. 
Simultaneous  use  (by  large  groups  of  students)  of  networked  desktop  simulations  for  such 
practice  would  shorten  the  number  of  hours  spent  on  actual  equipment,  with  equipment  being 
used  to  validate  rather  than  instruct.  Networked  simulations  also  could  support  the  integration  of 
on-the-job  learners  into  ongoing  resident  instruction.  An  accelerated  contracting  cycle  would 
enable  more  rapid  modification  to  existing  simulations,  thus  allowing  updates  to  simulation 
functionality  to  keep  pace  with  analogous  equipment  updates. 

Assessment  of  Discussion  Forums 

The  Sample  -  LandWarNet  eSignal  Technical  Forums  and  LandWarNet  Leaders  Forum 

Both  the  LandWarNet  eSignal  technical  forums  and  the  LandWarNet  Leaders  Forum 
were  assessed  as  part  of  this  research.  Discussion  forums  represent  the  significant  potential  of 
lifelong  learning  centers  to  foster  and  enable  culture  shift  toward  anytime,  anywhere 
participation  in  professional  self-development,  organizational  enculturation,  and  knowledge 
management.  Discussion  forums  also  play  a  key  role  in  fostering  the  development  of 
professional  learning  communities,  which  help  to  give  a  common  identity  to  distributed  learners. 
Examining  the  Leaders  Forum  provided  the  opportunity  to  explore  the  current  status  of  the  Fort 
Gordon  LLC  in  fostering  culture  shift  toward  the  use  of  discussion  forums  for  lifelong  learning 
and  horizontal  information  sharing.  Analyzing  the  eSignal  technical  forums  enabled  an 
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investigation  of  how  the  Fort  Gordon  LLC  supports  these  ongoing  efforts  by  distributed  Signal 
Soldiers. 

It  is  much  easier  to  build  a  collaborative  meeting  space  than  it  is  to  foster  transformative 
use  of  that  space  (Cianciolo,  Heiden,  &  Prevou,  2006;  Wenger,  McDermott,  &  Snyder,  2002). 
Yet,  user  activity  is  critical  to  achieving  the  organizational  impact  expected  to  result  from 
investments  in  collaborative  meeting  space.  For  this  reason,  assessing  the  status  of  the  discussion 
forums  component  required  going  beyond  the  impressive  number  of  forums  already  established 
in  eSignal  and  investigating  how  these  forums  were  used  and  to  what  extent  they  fostered 
knowledge  development  and  professional  growth.  The  development  of  social  and  intellectual 
capital  in  discussion  forums  is  a  critical  determinant  of  professional  growth  (Cianciolo  et  al., 
2006;  Lesser  &  Storck,  2001)  and  also  is  feasibly  (and  relatively  objectively)  measured,  so  it 
served  as  the  focus  of  the  present  research.  Four  aspects  of  social  and  intellectual  capital  growth 
defined  elsewhere  (Cianciolo  et  al.,  2006;  Dixon,  Allen,  Burgess,  Kilner,  &  Schweitzer,  2005) 
were  used  to  frame  assessment:  connections,  context,  content,  and  conversation. 

Assessment  Questions  &  Method 

The  analysis  of  discussion  forums  as  contributors  to  lifelong  learning  community 
development  focused  the  output  and  outcomes  associated  with  implementation  of  the  technical 
forums  and  Leaders  Forum.  Specifically,  the  following  questions  were  asked  and  the  associated 
metrics  were  used  to  focus  data  collection: 

o  Question:  Are  the  discussion  forums  being  actively  used? 

o  Metric:  %  of  registered  users  actively  using  the  system  (adoption  rate) 
o  Question:  Are  the  discussion  forums  supporting  the  development  of  interpersonal 
connections? 

o  Metric:  %  of  posts  that  contain  referrals  (to  self  or  others)  for  additional 
information  or  expertise 

o  Metric:  %  users  with  basic  biographical  information  viewable  by  others 
o  Question:  Are  the  discussion  forums  supporting  the  development  of  organizational 
context? 

o  Metric:  Presence  of  “built-in”  opportunities  to  participate  in  shared  experiences 
o  Question:  Are  the  discussion  forums  fostering  the  use  and  distribution  of  content? 

o  Metric:  %  of  knowledge  contributions  (i.e.,  downloads)  made  by  target  members 
(as  opposed  to  facilitators) 

o  Metric:  %  of  knowledge  contributions  (i.e.,  downloads)  with  contextualizing 
descriptions 

o  Question:  Are  the  discussion  forums  fostering  active  conversation? 

o  Metric:  %  of  discussion  contributions  made  by  target  members  (as  opposed  to 
facilitators) 

o  Metric:  %  of  initial  posts  followed  by  a  meaningful  response  within  24  hours  (i.e., 
not  just  an  acknowledgement) 

o  Metric:  %  of  conversation  threads  whose  posts  contain  references  to  one  another 

o  Metric:  %  of  incidents  of  unprofessional  commentary 
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To  assess  the  Fort  Gordon  LLC  discussion  forums,  sections  of  the  forums  themselves 
were  analyzed  directly.  For  the  Leaders  Forum,  the  discussion  and  knowledge-posting  activity  of 
particular  individuals  were  analyzed.  These  individuals  were  selected  based  on  their  class 
enrollment.  Beginning  in  the  Spring  of  2007,  students  in  the  Basic  Officer  Leadership  Course 
(BOLC),  the  Basic  Non-Commissioned  Officer  Course  (BNCOC),  and  the  Advanced  Non- 
Commissioned  Officer  Course  (ANCOC)  received  an  assignment  to  demonstrate  certain 
knowledge  management  proficiencies  using  the  Leaders  Forum.  These  proficiencies  included 
starting  a  discussion,  participating  in  a  discussion,  uploading  a  document  to  the  forum,  and 
editing  a  profile  containing  biographical  information.  The  intent  of  this  assignment  was  to  foster 
interest  in  and  ability  to  use  the  eSignal  technical  forums  and  the  BCKS  professional  forums. 
Students  who  had  received  the  assignment  and  who  had  several  months  to  complete  it  prior  to 
the  conduct  of  this  investigation  were  selected  to  analyze  the  rate  of  participation  in  the 
assignment  and  in  the  Leaders  Forum  generally.  The  total  number  of  students  selected  was  68 
(44  BOLC  students,  10  BNCOC  students,  and  14  ANCOC  students). 

In  the  case  of  the  eSignal  technical  forums,  conversations  in  particular  topic  areas  were 
selected  for  analysis  on  the  basis  of  their  presumed  relevance  to  the  forum  members  and  the 
present  research  effort.  These  topic  areas  included  notes  from  the  field  (including  lessons  learned 
from  Operation  Iraqi  Freedom  and  Operation  Enduring  Freedom),  technical  support  (for 
communications  systems  and  for  eSignal  itself),  and  specialty  areas  for  25B,  25N,  and  25S. 

Topic  areas  that  were  left  out  included  TRADOC  Lifelong  Learning  Center  (resource 
requirements  and  master  plans),  general  discussions  (e.g.,  high-level  conversation  on  information 
assurance,  LandWarNet,  LINUX,  etc.),  and  specialty  areas  not  including  25B,  25N,  and  25S 
(e.g.,  S6,  Warrant  Officer,  25U,  etc.).  The  number  of  conversations  analyzed  was  179, 
representing  1 8%  ( 1 79/1003)  of  the  number  of  conversations  currently  ongoing  at  the  time  of 
this  analysis  and  19%  of  the  number  of  discussion  posts  (810/4247). 

Summary  of  Findings 

Analysis  of  the  well-established  eSignal  technical  forums  revealed  steady  use  of  the 
forums  over  the  past  four  years,  which  have  been  populated  by  numerous  professional,  helpful 
conversations.  The  percentage  of  registered  users  participating  in  discussions  was  below  that 
described  as  typical  for  active  forums  (1-3%  compared  to  25-35%),  but  this  level  of  activity 
reflected  a  foundation  on  which  the  forums  can  be  built  further.  The  technical  forums  began  as 
mechanism  for  providing  technical  support  to  LLC  users,  but  have  evolved  to  meet  more  diverse 
needs.  Discussions  were  primarily  conducted  among  users  of  the  forums,  rather  than  by  the  LLC 
technical  staff,  suggesting  that  forum  members  felt  some  sense  of  ownership  over  the 
collaborative  space.  Levels  of  professionalism  were  very  high,  especially  in  an  unmoderated 
setting.  These  characteristics  suggest  that  active  cultivation  of  discussion  in  the  forums  would  be 
successful  in  further  developing  the  signal  learning  community. 

The  Leaders  Forum  represented  a  pilot  effort  for  exploring  how  to  cultivate  discussion 
and  other  activity  using  BCKS  as  the  online  collaborative  space.  The  focus  of  this  pilot  was  on 
establishing  the  mechanics  of  knowledge  management,  such  as  the  use  of  collaborative  software. 
During  the  course  of  this  investigation,  lessons  were  being  learned  about  what  sort  of  content 
could  stimulate  conversation  and  what  situational  characteristics  acted  to  motivate  Soldiers  to 
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use  online  discussions  to  communicate  with  each  other.  Activity  in  the  forums  reflected 
acquisition  of  the  “science”  of  knowledge  management,  but  the  development  and  implementation 
of  a  strategic  plan  for  integrating  existing  forums  into  the  knowledge  management  assignment 
would  enhance  the  cultivation  of  communities  of  lifelong  learners. 

Detailed  Findings 

Adoption  of  Forums.  To  explore  the  use  of  discussion  forums,  the  percentage  of  BOLC, 
BNCOC,  and  ANCOC  students  who  demonstrated  particular  proficiencies  in  Leaders  Forum  as 
outlined  in  their  knowledge  management  assignment  was  determined.  These  proficiencies,  as 
listed  above,  were:  (1)  start  a  discussion;  (2)  participate  in  a  discussion;  (3)  upload  a  document  to 
the  forum;  and  (4)  edit  biographical  information.  These  proficiencies  were  selected  from  a  larger 
list  (e.g.,  including  “know  the  difference  between  tacit  and  explicit  knowledge”  and  “download  a 
relevant  piece  of  knowledge”)  because  they  could  be  linked  directly  to  specific  individuals  and 
could  be  assessed  by  analyzing  forum  activity.  The  results  of  the  analysis  are  shown  below,  and 
include  only  those  students  who  were  registered  in  the  forum  [93%  (13/14)  of  the  ANCOC 
students  were  not  registered  in  the  forum  and  1 8%  (8/36)  of  the  BOLC  students  were  not 
registered]. 


Table  4.  i 

Knowledge  Management  Assignment  Completion  Rate 

Proficiency 

BOLC  (A  =36) 

BNCOC  (N  =  10) 

ANCOC  (N  =  1) 

Start 

discussion 

53%  (19/36) 

10%  (1/10) 

100%  (1/1) 

Participate  in 
discussion 

58%  (21/36) 

0%(0/10) 

0%  (0/1) 

Upload  file 

0%  (0/36) 

10%  (1/10) 

0%  (0/1) 

Edit  bio 

86%  (31/36) 

10%  (1/10) 

100%  (1/1) 

As  shown  in  the  table  above,  the  adoption  rate  of  the  Leaders  Forum  was  moderate. 
Although  ANCOC  student  participation  was  occasionally  at  100%,  this  sample  size  was  only  one 
student,  and  should  not  be  considered  a  representative  sample.  The  moderate  participation  rate 
was  apparently  driven  by  the  BOLC  students,  who  completed  the  assignment  at  a  significantly 
higher  rate  than  the  BNCOC  students.  Becoming  a  member  of  Leaders  Forum  is  one  of  the 
knowledge  management  proficiencies  that  was  not  assessed  because  it  was  unclear  whether 
students  were  responsible  for  registering  themselves  or  if  they  were  enrolled  as  a  group  by 
someone  else.  To  the  extent  that  registration  was  voluntary,  adoption  rates  were  best  for  the 
BNCOC  students  (100%).  Moreover,  all  of  the  BNCOC  students  were  cross-registered  as 
members  of  NCO  Net  -  another  BCKS  forum.  High  percentages  of  both  BNCOC  students  and 
ANCOC  students  [90%  (9/10)  and  93%  (13/14),  respectively]  were  potentially  cross-registered 
as  members  in  the  eSignal  technical  forums.8  Three  of  the  44  BOLC  students  (7%)  were  cross- 
registered  in  other  BCKS  forums  and  27%  (12/44)  were  potentially  cross-registered  in  the 


8  Cross-registration  in  the  eSignal  technical  forums  was  determined  by  searching  the  technical  forum  member  list  for 
the  names  of  the  BOLC,  BNCOC,  and  ANCOC  students.  The  technical  forums  member  list  did  not  include 
biographical  data,  so  where  common  names  (e.g.,  Ralph  Jones)  were  found,  it  could  not  be  verified  for  certain 
whether  the  student  name  and  technical  forum  member  name  belonged  to  the  same  person. 
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eSignal  technical  forums.  None  of  the  students  in  any  course  made  content  or  conversation 
contributions  in  the  e-Signal  technical  forum  discussions  analyzed. 

The  relatively  high  frequency  of  forum  cross-membership  among  the  BNCOC  and 
ANCOC  students  suggests  that  limited  activity  among  Leaders  Forum  members  was  not 
necessarily  due  to  unwillingness  to  access  knowledge  management  resources.  An  alternative 
explanation  is  that  the  number  of  passionate,  active  participants  in  an  online  community  typically 
represents  a  small  proportion  (25-35%)  of  the  member  population  (Wenger  et  al.,  2002).  At  least 
in  the  case  of  the  BOLC  students,  member  activity  well  exceeded  this  range.  Examination  of  the 
posts  made  by  BOLC  discussion  participants,  however,  indicated  that  approximately  36% 

(28/77)  of  the  contributions  were  made  primarily  to  complete  the  assignment  requirement,  rather 
than  to  engage  the  professional  community  and  conduct  self-development.  For  example,  one  of 
these  posts  said  “I’m  trying  this  out  for  my  homework.”  A  chain  of  posts  following  an  initial  post 
said  “interesting,”  then  “really  interesting,”  then  “seriously  interesting.”  It  appeared  that  the 
“science”  of  knowledge  management  was  successfully  acquired  by  these  students,  as  expected, 
but  that  the  pilot  forum  was  not  yet  a  ready  canvas  for  the  “art”  of  knowledge  management. 

A  second,  more  likely  alternative  explanation  is  that  the  students  had  other,  more 
effective  means  for  sharing  information  with  their  peers.  First,  the  students  were  co-located  at 
Fort  Gordon,  so  much  of  their  communication  could  happen  face-to-face  instead  of  online. 
Second,  limited  public  computing  facilities  on  post  reduced  the  ease  with  which  students  could 
actively  engage  in  online  discussions  outside  of  the  classroom  when  they  were  not  co-located. 
Third,  participation  in  Leaders  Forum  was  part  of  an  assignment  and  not  a  natural  outgrowth  of 
interest  in  what  the  forum  had  to  offer.  The  pilot  Leaders  Forum  purposefully  was  established 
largely  devoid  of  content  in  order  to  serve  as  a  place  for  demonstrating  and  cultivating  the 
collaborative  knowledge  development  processes.  However,  the  initial  content  in  a  forum  is 
critical  for  engaging  members  in  active  participation  (see,  e.g.,  Dixon  et  al.,  2005).  Moreover,  it 
appeared  that  the  content  needs  of  the  Leaders  Forum  target  audience  could  be  addressed  by 
visiting  other  forums,  such  as  NCO  Net  or  PlatoonLeader.mil. 

At  the  time  of  conducting  these  analyses,  there  were  56,154  registered  users  in  the 
eSignal  technical  forums.  Of  this  number,  299  unique  members  contributed  to  the  conversations 
sampled  as  part  of  this  investigation,  representing  less  than  1%  of  the  membership.  Additional, 
unique  forum  members  would  almost  certainly  be  represented  if  all  of  the  conversations  had 
been  sampled,  but  by  the  same  token,  other  members  who  participated  in  the  sampled 
discussions  would  not.  Assuming  that  the  same  ratio  of  unique  contributors  to  conversation  posts 
found  in  the  analyzed  conversations  [37%  (299/810)]  also  characterized  the  unanalyzed 
conversations,  then  approximately  3%  (1,571/56,154)  of  registered  users  contributed  to  ongoing 
conversations.  This  percentage  is  less  than  the  25-35%  active  participation  rate  cited  in  Wenger, 
et  al  (2002).  In  the  discussions  analyzed,  roughly  the  same  number  of  posts  were  made  in  2004, 
2005,  2006,  and  2007,  which  indicated  a  steady  trend  of  conversation  activity  in  the  technical 
forums. 


Connections.  Seventy  percent  (33/47)  of  the  Leaders  Forum  members  (representing 
primarily  BOLC  students,  N=  31)  edited  their  biographical  information  beyond  the  basic 
required  information  for  the  registering  in  the  forum.  The  required  biographical  information 
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included  rank,  branch,  duty  status,  profile  (name  and  email),  key  words  (to  be  used  as  search 
terms  when  finding  people  in  the  forum),  and  willingness  to  share  knowledge  (true/false).  Non- 
required  information  included  19  additional  data  entry  fields,  including  spoken  languages, 
expertise,  theater  deployments,  reason  for  joining  the  forum,  etc.  The  middle  column  of  Table  5 
below  shows  the  percentage  of  forum  members  who  filled  out  each  of  these  additional  fields. 


Table  5.  Percentage  of  Non-required  Data  Fields  Filled  Out 


Non-required 

Data  Entry  Field 

Percentage  Data  Entry,  Non- 
Required  Biographical  Information 
(Total  N =47) 

Percentage  Data  Entry, 
All  Data  Entry  Fields 
(Total  N  =  47) 

Middle  Initial 

74%(JV=35) 

74%  (A  =35) 

Reason  For  Joining  Forum 

49%  (N-  23) 

40%  (/V=  19) 

Mobile  Phone 

49%  (N=  23) 

49%  (N  =23) 

Functional  Area/MOS 

45%  ( =21) 

47%  (N  =  22) 

Education 

oo 

II 

00 

m 

38%  (N=  18) 

Grade/Rank  or  Title 

38%  (N=  18) 

62%  (W  =29) 

Spoken  Languages 

36%  (N-  17) 

36%  (N=  17) 

Work  Phone 

23%  (N=  1 1) 

23%  (N=  1 1) 

Job  Experience 

21  %(N=  10) 

21  %{N=  10) 

Theater  Deployments 

21%  (N=  10) 

23%(/V=  11) 

DSN 

£ 

II 

\5%{N=1) 

ASI 

9%  (jV  =  4) 

9%{N=A) 

Fax 

6%  (N=  3) 

6%(/V=3) 

Expertise/Competencies 

4%  (/V  =  2) 

13%  (N=  6) 

Additional  Background 

2  %(N=  1) 

26%  (N=  12) 

Title 

0%(/V=0) 

47%(W=22) 

Homepage 

0%  (N  =  0) 

0%(/V=0) 

Fax 

/ — \ 

O 

II 

y 

o 

0%  (N  =  0) 

ICQ  Number 

0%  (N  =  0) 

0%  (N  =  0) 

As  shown  in  this  column,  14  of  the  1 9  non-required  biographical  information  fields  were 
used.  Of  these  fields,  the  percentage  filled  out  ranged  from  2%  to  74%,  with  the  majority  (58%) 
of  percentages  being  1 5%  or  higher.  Notably,  the  Expertise/Competencies  field  only  had  a  4% 
fill  rate,  whereas  Middle  Initial  field  had  a  74%  fill  rate.  Alone,  however,  these  unanalyzed 
percentages  did  not  accurately  reflect  the  degree  to  which  the  Leaders  Forum  pilot  facilitated  the 
growth  of  social  connections. 

First,  the  user  audience  should  be  considered  when  interpreting  what  the  above  metrics 
mean.  For  example,  the  majority  of  the  Leaders  Forum  members  in  the  present  sample  (46/47) 
were  near  the  beginning  of  their  Army  careers  and  so  likely  did  not  have  theater  deployments  or 
expertise  to  report.  The  relatively  detailed  background  and  expertise  description  provided  by  the 
single  ANCOC  forum  member  relative  to  the  limited  descriptions  provided  by  the  BOLC  and 
BNCOC  students  suggests  this  is  a  possibility.  Second,  some  of  the  important  information  asked 
for  in  the  required  fields  had  been  entered  into  the  required  Keywords  field,  including  entries  for 
MOS,  expertise,  and  additional  background.  When  these  entries  were  taken  into  account,  the 
percentage  of  forum  members  providing  information  increased.  Third,  the  nature  of  the 
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information  provided  in  the  non-required  data  entry  fields  should  be  considered.  For  example, 
30%  of  the  additional  information  provided  in  the  required  Keywords  field  was  unrelated  to  the 
business  of  the  forum  (e.g.,  “Married”  and  “thorough  and  open-minded”).  Removing  these 
entries  reduced  the  percentage  of  forum  members  providing  information.  The  revised 
contribution  rates  are  shown  above  in  Table  6  above  in  the  right-hand  column.  These  revised 
percentages  reflect  a  moderate  degree  of  facilitation  of  social  network  growth,  but  it  should  be 
noted  that  the  majority  of  entries  in  the  biographical  information  fields  were  minimal  (e.g.,  one- 
word  entries).  Here  too,  it  appears  that  the  students  using  the  Leaders  Forum  effectively  learned 
the  mechanics  of  knowledge  management,  which  was  a  goal  of  this  pilot  forum,  but  that  the 
development  of  professional  community  would  require  additional  effort. 

None  of  the  user  profiles  in  the  technical  forums  contained  biographical  information.  The 
design  of  the  technical  forum  user  profile  did  not  permit  the  entry  or  display  of  such  information. 

Another  reflection  of  social  network  development  may  be  found  in  the  referrals  that 
discussants  make  to  themselves  or  others  who  could  provide  additional  information  or  expertise. 
None  of  the  posts  in  the  Leaders  Forum  referred  conversation  participants  to  others.  Thirty-eight 
percent  of  forum  members  sampled  indicated  that  they  were  unwilling  to  share  information, 
although  it  should  be  noted  that  it  is  unclear  whether  this  selection  was  a  conscious  choice  or  the 
unintentional  acceptance  of  a  default  setting  (e.g.,  some  members  who  selected  “false”  to 
characterize  their  willingness  to  share  information  provided  contact  information).  In  the  sampled 
discussions  in  the  technical  forum,  1 1%  of  posts  referred  others  to  the  poster  or  some  other 
person  for  additional  expertise  or  information. 

Context.  Both  the  eSignal  technical  forums  and  the  Leaders  Forum  were  examined  for 
the  presence  of  “built-in”  opportunities  to  stimulate  discussion.  Such  opportunities  stand  out  in 
contrast  to  “organic”  conversations  that  arise  out  of  regular  activity  in  the  forum.  Instead,  built-in 
opportunities  are  organized  events  such  as  digital  stories,  book  reviews,  collective  exercises,  and 
the  like,  which  are  released  in  the  forum  purposefully  to  generate  a  shared  experience  and  get 
conversation  going  (Cianciolo,  Cianciolo,  Prevou,  &  Morris,  2007).  Built-in  opportunities  for 
discussion  may  be  contributed  by  facilitators  and  forum  members  and  help  to  build  professional 
community  and  stimulate  the  exchange  of  tacit  knowledge.  Neither  the  technical  forums  nor  the 
Leaders  Forum  included  these  manufactured  opportunities.  The  technical  forums  did  include  a 
small  handful  of  polls,  which  may  be  used  to  give  members  a  sense  of  their  community 
demographics  and  opinions,  but  without  discussion  these  do  not  serve  as  a  way  to  generate 
shared  experience. 

It  should  be  noted  that  in  some  sense  the  technical  forums  were  not  designed  to  facilitate 
the  development  of  professional  community,  but  more  to  serve  as  information  portals  for  people 
with  explicit  technical  questions.  It  may  be  the  case  that  the  purpose  of  these  forums  is  not 
served  by  introducing  built-in  opportunities  for  discussion  because  users  have  opportunities  to 
develop  professional  community  in  other  forums,  such  as  NCO  Net,  PlatoonLeader.mil,  and 
CompanyCommand.mil.  Although  in  a  pilot  stage,  the  Leaders  Forum  ultimately  is  intended  to 
foster  the  development  of  community  and  knowledge  sharing  skills,  so  these  engineered 
opportunities  will  be  useful  for  stimulating  activity  and  shared  experience. 
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Content.  Two  metrics  were  assessed  to  explore  the  quality  of  the  content  in  the  forums. 
These  metrics  were  selected  to  determine  the  relative  activity  of  forum  members  compared  to 
facilitators  in  posting  content  and  the  degree  to  which  content  posted  to  the  forum  was 
actionable.  Although  it  is  important  for  facilitators  to  post  content  to  a  forum,  it  is  an  important 
sign  of  community  health  that  forum  members  share  the  responsibility  for  disseminating 
information  and  solutions.  Without  this  kind  of  balance,  the  forum  becomes  a  knowledge 
repository  designed  in  a  top-down  fashion  rather  than  an  emergent  body  of  knowledge  created 
through  professional  exchange.  Actionable  content  is  content  (e.g.,  a  training  plan,  a 
presentation,  a  standard  operating  procedure,  etc.)  that  is  supported  with  contextualizing 
information.  Contextualizing  information  is  critical  for  potential  users  to  understand  how  and 
when  such  content  may  be  applied  (Dixon  et  al.,  2005)  and  includes  descriptions  of  what  the 
content  is,  when  it  was  created,  how  it  was  used,  who  used  it,  and  so  on. 

In  the  Leaders  Forum  sample  studied,  one  forum  member  posted  content,  but  it  was  not 
coupled  with  contextualizing  information.  In  the  technical  forums  no  content  uploads  were  made, 
although  several  posts  included  pasted  content  from  external  sources  or  links  to  external  sources. 
More  in-depth  analysis  of  content  quality  was  not  conducted  because  relatively  few  contributions 
were  made  by  the  samples  studied. 

Conversation.  Four  metrics  were  used  to  assess  conversation  activity  and  quality. 
Conversation  activity  is  reflected  in  the  length  of  time  between  posts,  particularly  posts  that 
initiate  a  discussion,  asking  for  help  or  opinions.  These  posts  should  be  followed  up  with  some 
kind  of  acknowledgement  and  initial  comment  within  24  hours  in  order  to  foster  confidence  in 
forum  users  that  the  forum  can  meet  their  needs.  Conversation  quality  is  reflected  in  the  degree 
to  which  conversations  have  a  professional  tone,  reflect  the  active  participation  of  the  community 
(as  opposed  to  just  facilitators),  and  feature  links  to  one  another  that  represent  exchange  among 
professionals.  Some  additional  content  analysis  based  on  the  rubric  presented  in  Cianciolo  et  al., 
(2006)  was  performed.  The  purpose  of  this  analysis  was  to  investigate  the  utility  and 
effectiveness  of  the  conversation. 

Of  the  1 79  initiating  discussion  posts  analyzed  in  this  investigation,  44%  were  responded 
to  within  24  hours,  reflecting  a  relatively  low  rate  of  conversation  activity  (90%  or  more  is 
ideal).  Less  than  1%  of  these  posts  (N=  6)  had  an  unprofessional  tone,  reflecting  a  high  level  of 
professionalism  in  this  unmoderated  forum.  Unprofessional  tone  was  defined  as  saying  negative 
things  about  another  person  or  group  of  people.  Twelve  percent  (97/810)  of  the  discussion  posts 
referred  to  other  discussion  posts  in  a  thread.  In  the  discussions  analyzed,  participants  seemed  to 
make  an  active  effort  to  summarize  or  clarify  the  discussion  by  referring  to  what  others  said. 
Seventeen  percent  (N=  138)  of  posts  were  made  by  facilitators,  which  indicated  that  the  large 
majority  of  ongoing  discussion  was  conducted  by  forum  members. 

To  analyze  conversation  utility  and  effectiveness,  discussion-initiating  posts  were  first 
separated  according  to  whether  they  received  a  response  (no  response  =  1  post-thread;  1  or  more 
responses  =  multi-post  thread).  Next,  the  initiating  posts  were  categorized  into  types, 
corresponding  to  the  post  type  definitions  provided  in  Cianciolo  et  al.,  (2006).  These  post  types 
were:  Invitation  to  Build  “New”  Knowledge,  Request  for  Input,  Directed  Question,  Request  for 
Knowledge  Resources,  Request  for  Expert,  Information  Post,  and  Unknown.  Two  additional  post 
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types,  Rant  and  Inferred  Question,  were  added  for  the  present  analysis.  A  Rant  was  defined  as  a 
post  that  points  out  deficiencies  of  some  kind  in  a  process,  policy,  or  procedure  but  that  is  not 
explicitly  seeking  feedback  or  broader  opinion.  An  Inferred  Question  was  defined  as  a  post  that 
does  not  explicitly  ask  a  question  but  that  indicates  the  poster  is  having  a  problem  of  some  kind, 
perhaps  a  form  or  requesting  help  that  is  typical  of  technical  support  forums  (e.g.,  “I  click  the 
link  and  it  goes  to  the  wrong  page.”).  Table  6  below  shows  the  breakdown  of  initiating  post  type 
within  both  types  of  thread  (1-post  and  multi-post). 

As  shown  in  the  table,  15%  (26/179)  of  the  initiating  posts  asking  questions  did  not 
receive  a  follow-up  post  in  the  forum.  The  majority  (92%)  of  these  posts  were  inferred  questions, 
requests  for  experts,  and  requests  for  resources.  The  majority  of  initiating  posts  were  followed  by 
a  response.  The  average  number  of  posts  in  a  multi-post  discussion  thread  was  4,  ranging  from  1 
to  55.  Directed  questions  and  requests  for  input  were  especially  effective  at  generating  responses 
in  the  forum.  Requests  for  resources  also  generated  a  response  more  often  than  not,  although 
requests  for  experts  did  not. 


Table  6.  Type  of  Initiating  Posts  for  One-Post  and  Multi-Post  Threads 


Post  Type 

#  Initiating 

1  -Post  Threads 

%  of  Total 

#  Initiating  Multi- 
Post  Threads 

%  of  Total 

Directed  Question 

0 

0% 

55 

36% 

Invitation  to  Build  “New” 
Knowledge 

1 

4% 

1 

1% 

Information  Post 

0 

0% 

9 

6% 

Inferred  Question 

11 

42% 

16 

11% 

Request  for  Input 

0 

0% 

39 

25% 

Request  for  Resources 

4 

15% 

19 

12% 

Request  for  Expert 

9 

35% 

5 

3% 

Rant 

1 

4% 

6 

4% 

Unknown 

0 

0% 

3 

2% 

Total 

26 

100% 

153 

100% 

The  utility  or  effectiveness  of  a  multi-post  thread  was  determined  by  applying  the  criteria 
outlined  in  Cianciolo  et  al.  (2006)  for  each  type  of  initiating  post.  The  criteria  capture  whether 
the  multiple  posts  in  a  thread  adequately  addressed  the  question  raised  (or  information  shared)  by 
the  initiating  post.  For  example,  the  initiating  post  types  of  Invitation  to  Build  “New” 

Knowledge,  Request  for  Input,  Rant,  and  Information  Post  were  examined  for  the  degree  to 
which  they  stimulated  participation  among  the  broader  population.  The  initiating  post  types  of 
Directed  Question,  Request  for  Expert,  Request  for  Knowledge  Resources,  and  Inferred  Question 
were  examined  for  the  degree  to  which  the  question  posed  was  actually  answered  inside  the 
forum.  Using  these  criteria,  77%  (137/179)  of  the  initiating  posts  were  followed  by  useful  and 
effective  conversation. 
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Recommendations 


The  technical  forums  have  a  solid  foundation  on  which  to  build  further  participation  and 
to  stimulate  the  evolution  of  the  forums’  purpose  from  technical  support  to  a  professional 
development  learning  community.  Activities  that  may  increase  participation  in  the  technical 
forums  should  enhance  awareness  of  ongoing  activity  in  the  forum,  to  include  featuring  new  and 
hot  conversations  on  the  main  eSignal  page  and  encouraging  members  to  use  the  forum  for 
professional  exchange.  LLC  staff  members  who  serve  as  forum  facilitators  could  further  fuel 
activity  by  conducting  periodic  needs  analyses  and  addressing  these  in  the  forum  with  “guest” 
discussants. 

The  Leaders  Forum  pilot  reflects  the  critical  recognition  that  knowledge  management 
competencies  are  an  important  determinant  of  lifelong  learning.  The  recommendations  presented 
here  are  intended  to  help  the  Fort  Gordon  LLC  develop  both  the  “art”  and  “science”  of  the 
horizontal  information  sharing  process.  Two  key  lessons  learned  that  should  be  carried  away 
from  the  current  status  of  the  pilot  effort  are  (1)  content  plays  a  key  role  in  stimulating  discussion 
and  community  development;  and  (2)  people  use  the  most  expedient  mode  to  communicate  with 
one  another.  Greater  success  in  stimulating  culture  shift  and  adoption  of  knowledge  management 
would  be  achieved  by  using  the  knowledge  management  assignment  to  guide  students’ 
interactions  with  well-established  forums  that  connect  them  with  the  larger,  distributed 
professional  community. 

In  order  to  reduce  the  likelihood  that  unproductive  conversations  are  conducted  by  novice 
students  in  the  professional  community,  the  assignment  could  instruct  students  to  pull  content 
from  the  established  forums  and  then  distribute  and  discuss  it  in  the  Leaders  Forum.  Consistent 
with  the  current  approach,  such  facilitation  could  be  accomplished  in  a  classroom  setting,  and 
could  be  implemented  as  a  one-day  seminar  when  students  are  exposed  to  and  enrolled  in  the 
appropriate  BCKS  forums.  To  stimulate  participation,  the  students  could  be  given  class  time  to 
explore  the  BCKS  forums,  to  use  the  Leaders  Forum  to  share  what  they  learned,  and  to  have  live 
conversations  about  items  of  particular  interest  they  discovered.  Increased  levels  of  instructor 
moderation  of  the  Leaders  Forum  would  also  encourage  discussion  as  well  as  ensure  that  the 
culture  of  knowledge  management  is  understood  and  adopted  by  the  students. 

Assessment  of  Leader  Education 
The  Sample  -  Signal  Captains '  Career  Course  (SCCC) 

The  SCCC  serves  as  a  representative  example  for  exploring  how  leader  education  can  be 
conducted  using  LLCs.  Leader  education  is  a  component  of  most  proponent  schoolhouse  course 
offerings,  so  it  is  useful  to  understand  how  the  technologies  that  constitute  an  LLC  not  only 
permit  the  cost-effective,  wide-reaching  delivery  of  course  materials  but  also  enable  equivalent 
learning  experiences  across  diverse  learning  environments.  An  initial  analysis  of  LLC  impact  on 
leader  education  (intermediate-level  education  for  field  grade  officers)  was  presented  in 
Cianciolo  (2007),  but  in  that  study  greater  emphasis  was  placed  on  the  equivalence  of  learning 
content  vice  equivalence  of  learning  experience  or  environment.  Although  the  same  tasks  and 
standards  may  be  applied  to  both  classroom  and  blended  modes  of  education  delivery, 
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differences  in  instructional  strategies  and  learning  contexts  may  moderate  educational  impact 
(Clark,  1994;  Firdyiwek,  1999).  The  potential  for  cost  savings  associated  with  delivering  blended 
instruction,  such  as  reduced  housing  and  student  compensation  costs,  is  widely  recognized.  In  the 
present  research,  it  was  necessary  to  explore  the  application  of  metrics  that  capture  instructional 
strategy  and  learning  experience  and  their  impact  on  learning  performance. 

Broadly  speaking,  the  SCCC  prepares  signal  captains  to  lead  a  company-sized  signal  unit 
and  to  serve  as  the  signal  officer  (S6)  on  a  combined  arms  staff  (see,  e.g.,  North,  2006).  The 
recently  revised  SCCC  consists  of  a  TRADOC-prescribed  common  core  and  seven  instructional 
modules  that  cover  general  knowledge  (e.g.,  general  leader  skills,  personnel  management, 
logistics,  etc.),  signal  theory,  information  technology,  information  management,  Department  of 
Defense  communications,  network  management,  and  combined  arms  planning  (North,  2006).  In 
the  resident  version  of  the  course,  these  topics  are  covered  via  a  combination  of  face-to-face 
lecture,  hands  on  training,  practical  exercise,  and  some  computer-based  instruction.  In  the 
blended  version  of  the  course,  the  common  core  and  seven  modules  are  allocated  to  five  phases 
of  instruction-three  of  which  are  individual  distance  learning  (Phases  I,  II,  and  IV)  and  two  are 
resident  phases  held  at  the  Signal  Center  (Phases  III  and  V)-and  a  single  block  of  self¬ 
development. 

Assessment  Questions  and  Method 

In  the  present  research,  outputs  and  outcomes  associated  with  equivalence  of  resident 
SCCC  and  blended  (distance  learning  +  resident)  SCCC  were  explored  in  order  to  determine  the 
educational  impact  of  delivering  leader  education  via  a  combination  of  resident  and  distance 
learning  (DL)  formats.  Specifically,  the  following  questions  were  asked: 

o  Question:  Does  administering  the  SCCC  online  enable  uniform  access  to  training 
content? 

o  Metric:  Content  equivalence  across  resident  and  online  learners 
o  Metric:  Instructional  strategy  equivalence  across  resident  and  online  learners 
o  Question:  Does  administering  the  SCCC  online  produce  uniform  learning  performance 
across  resident  and  online  learners? 

o  Metric:  Performance  equivalence  across  resident  and  online  learners 

A  combination  of  methods  was  used  to  explore  these  questions,  including  interviews, 
archival  data  analysis,  focus  groups,  and  classroom  observation.  Interviews  were  conducted  with 
two  instructors  who  oversaw  SCCC-blended,  including  administration,  content  delivery,  and 
performance  evaluation  during  both  distance  learning  (DL)  and  resident  phases  of  the  course. 
One  of  these  instructors  also  conducted  performance  evaluation  during  the  capstone  exercises  of 
SCCC-resident.  Brief,  informal  discussions  also  were  held  with  two  SCCC  course  developers 
and  a  more  in-depth  discussion  was  held  with  an  Army  Training  and  Doctrine  Command  quality 
assurance  officer  who  was  concurrently  evaluating  the  content  equivalence  of  SCCC-resident 
and  SCCC-blended.  Overall,  the  purpose  of  these  interviews  was  to  determine  the  nature  of 
instruction  in  the  two  versions  of  the  SCCC. 
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Archival  data  analysis  consisted  of  examination  of  the  SCCC-resident  and  SCCC-blended 
programs  of  instruction  and  course  “crosswalk”  diagrams.9  Also  studied  was  the  course  map  for 
SCCC-blended.  The  purpose  of  the  archival  data  analyses  was  to  explore  the  content 
equivalence  of  SCCC-resident  and  SCCC-blended.  An  additional  purpose  was  to  determine  the 
different  modes  of  instruction  (e.g.,  hands  on  training  vs.  equipment  simulation)  used  to  deliver 
SCCC  content  (i.e.,  instructional  strategy  equivalence). 

Two  focus  groups  (N=7  and  N=  8)  were  conducted  with  SCCC-blended  students.  The 
purpose  of  these  focus  groups  was  to  explore  further  instructional  strategy  equivalence.  The 
questions  asked  of  focus  group  participants  centered  on  their  experiences  during  the  distance 
learning  phases  of  the  course.  Specifically,  the  questions  asked  about  (a)  interactivity  among 
students  and  with  the  instructor  while  distributed;  (b)  usability/access  issues  associated  with 
online  content  delivery;  (c)  impressions  of  content,  delivery,  and  learning  achieved;  (d)  tradeoffs 
involved  in  completing  coursework  at  home  vs.  at  a  schoolhouse;  and  (e)  participation  in  the 
block  of  voluntary  self-development  courseware. 

Classroom  observations  were  conducted  during  the  combined  arms  staff  exercise  held  at 
the  end  of  both  SCCC-resident  and  SCCC-blended.  In  the  combined  arms  staff  exercise, 
students  role-played  the  members  of  a  Stryker  Brigade  Combat  Team  staff  and  conducted  the 
military  decision-making  process  to  form  a  plan  for  addressing  stability  operations  in 
Azerbaijan.  The  mission  analysis  brief  and  course  of  action  brief  of  two  groups  of  SCCC- 
blended  students  and  one  group  of  SCCC-resident  students  were  assessed  using  the  Tactical 
Thinking  Behaviorally  Anchored  Rating  Scales  (Phillips,  Ross,  &  Shadrick,  2006)  as  a  guiding 
framework.  Specifically,  the  briefs  were  assessed  for  the  stage  of  cognitive  development 
(novice,  advanced  beginner,  competent,  proficient,  and  expert)  reflected  in  the  slides  presented 
and  associated  discussion.  General  observations  were  written  down  and  then  collectively 
associated  with  a  stage  of  development. 

It  should  be  noted  that  the  sample  of  SCCC  student  groups  interviewed  and  observed  may 
not  be  representative  and  that  a  larger  sample  is  desirable  for  drawing  strong  conclusions.  The 
findings  based  on  this  sample  are  consistent  with  previous  research,  so  it  is  possible  to  conclude 
that  this  sample  is  not  an  outlier. 

Summary  of  Findings 

The  present  analysis  indicated  that  blended  learning  solutions,  partially  hosted  in  the  Fort 
Gordon  LLC,  enabled  roughly  uniform  distribution  of  leader  education  content  across  learner 
types.  Variation  in  learning  experience  across  each  type  of  learner  was  identified,  however  this 
dissimilarity  did  not  appear  to  create  meaningful  differences  in  performance  during  the  capstone 
exercise  observed.  Student  groups  having  received  both  types  of  instruction  demonstrated  novice 
levels  of  performance,  with  slightly  more  advanced  cognitive  development  demonstrated  by  the 
resident  group.  The  blended  group  did  require  coaching,  however,  in  order  to  match  the  level  of 
performance  demonstrated  by  the  resident  group. 


9  Crosswalks  compare  both  versions  of  the  course,  showing  areas  of  shared/unique  topic  coverage  and 
similarities/differences  in  recommended  hours  for  completion. 
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Differences  in  learning  experience  stemmed  primarily  from  a  lack  of  interactivity  among 
the  student  and  his  or  her  instructors  and  peers  during  the  DL  phases  of  blended  instruction  and 
from  a  lack  of  incentives  and  processes  in  place  to  reward  effective  self-instruction  strategies  for 
distance  learners.  As  noted  in  Leonard,  Winkler,  Hove,  et  al.  (2001),  implementing  procedures 
for  resourcing,  supporting,  and  encouraging  at-home  instruction  is  a  critical  contributor  to  the 
success  of  DL.  Although  the  performance  implications  associated  with  unequal  learning 
experiences  were  not  seen  in  the  present  research,  learner  dissatisfaction  with  DL  may  foster 
negative  attitudes  towards  the  capability  of  the  Army  to  train  effectively  and  the  Army’s 
motivations  for  providing  training  at  home.  Negative  attitudes  such  as  these  may  reduce  affective 
organizational  commitment  and  individual  readiness. 

Detailed  Findings 

Equivalence  of  Course  Content.  Examination  of  the  program  of  instruction  for  both 
versions  of  the  SCCC  largely  indicated  course  content  equivalence.  That  is,  the  same  course 
content  was  made  available  to  all  learners,  regardless  of  location,  and  involved  roughly  the  same 
amount  of  recommended  hours  (760  hours  for  resident,  670.5  hours  for  blended)10.  Differences 
in  course  content  actually  received  arose  when  (a)  there  were  usability  or  access  problems  with 
the  distance  learning  technologies;  and  (b)  required  components  in  the  resident  version  of  the 
course  were  voluntary  in  the  blended  version  of  the  course. 

First,  interviews  with  instructors  revealed  that  the  TRADOC-prescribed  common  core 
component,  administered  online  for  students  taking  the  blended  version  of  the  course,  had  so 
many  technical  problems  that  many  students  have  been  allowed  to  graduate  without  having 
completed  the  common  core.  The  TRADOC  prescribed  common  core  accounts  for  64  hours  of 
instruction  ( 1 8  topics),  covering  branch-general  topics  such  as  equal  opportunity,  law  of  war,  and 
information  assurance.  Resident  students,  in  contrast,  take  the  common  core  via  computer-based 
instruction  delivered  in  the  schoolhouse,  avoiding  online  delivery  problems.  For  this  reason, 
resident  students  complete  this  component  of  the  course  whereas  blended  students  may  not. 
Students  interviewed  in  focus  groups  most  frequently  reported  that  they  had  begun  the  SCCC 
prior  to  its  conversion  to  the  five-phase  format  in  2006,  so  they  did  not  take  the  common  core 
online.  Of  two  interviewees  who  did  report  taking  the  common  core,  one  reported  significant 
technical  difficulty  and  one  did  not.  Focus  group  participants  also  reported  technical  difficulties 
when  accessing  other  online  portions  of  the  course,  but  these  problems  did  not  prevent  them 
from  completing  the  coursework. 

Second,  some  components  of  the  seven  instructional  modules  were  taught  in  a  classroom 
setting  for  resident  students  whereas  SCCC-blended  students  received  these  same  components  as 
voluntary  self-development  (totaling  about  161  hours  of  instruction).  It  may  be  the  case  that 
distance  learning  students  complete  the  voluntary  self-development,  but  given  the  large  number 


10  Differences  in  hours  between  the  two  versions  of  SCCC  do  not  reflect  the  subtraction  of  a  particular  annex  or 
annexes  of  instruction.  Rather,  small  differences  accrue  over  content  areas  such  that  although  SCCC-blended  has 
more  recommended  hours  for  some  topics,  SCCC-resident  overall  has  a  greater  number  of  recommended  hours.  In 
general,  SCCC-blended  topics  taught  in  residence  have  several  fewer  (-40  hours  per  topic)  recommended  hours  than 
the  same  topics  taught  in  SCCC-resident.  This  difference  enables  SCCC-blended  students  to  return  more  quickly  to 
their  civilian  jobs  and  families. 
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of  hours  necessary  to  complete  the  required  distance  learning  coursework  in  the  context  of 
civilian  job  and  Family  demands,  it  is  likely  that  few  actually  do.  Focus  group  participants 
unanimously  reported  that  they  did  not  complete  any  of  the  self-development  coursework.  Some 
of  the  people  interviewed  did  not  know  that  such  coursework  was  available. 

For  these  two  reasons,  differences  in  course  content  actually  received  represented 
somewhere  between  161-225  hours  of  instruction  and,  more  importantly,  16-34  content  topics.  It 
should  be  noted  that  prior  to  the  introduction  of  online  instruction,  the  non-resident  portion  of  the 
blended-SCCC  was  conducted  via  mail  correspondence.  The  course  was  conducted  in  two 
phases  instead  of  five,  with  the  second  phase  conducted  in  residence  at  Fort  Gordon  over  two 
weeks.  Blended-SCCC  students  received  a  great  deal  less  content  than  resident  students,  and 
what  content  they  received  was  strictly  text-based  without  interactivity.  In  focus  groups  and 
interviews,  this  educational  context  was  reported  as  less  preferable  by  students  and  course 
developers  alike.  Lack  of  equivalence  in  course  content  among  the  two  current  versions  of  the 
SCCC  should  be  considered  in  this  light. 

Equivalence  of  Instructional  Strategy.  The  application  of  an  effective  instructional 
strategy  that  leverages  technical  capability  is  critical  for  achieving  the  educational  impact  made 
possible  via  blended  learning  solutions  for  leader  training  (Clark,  1994;  Firdywik,  1999).  In  the 
case  of  blended  learning,  the  conditions  must  be  set  for  effective  self-instruction  strategy. 
Leonard,  Winkler,  Hove,  et  al.  (2001)  outlined  five  methods  for  ensuring  the  effective 
implementation  of  blended  learning,  as  follows: 

1 .  Carefully  selecting  segments  of  instruction  amenable  to  instructorless  or  distributed 
learning 

2.  Using  appropriate  and  relevant  instructional  media 

3.  Ensuring  that  sufficient  changes  are  made  to  existing  processes  and  support  activities 
(e.g.,  to  enable  instructor  support  for  online  learners) 

4.  Providing  adequate  resources  to  implement  blended  learning  (e.g.,  technology,  technical 
support  personnel) 

5.  Providing  sufficient  incentives  for  students,  commanders,  and  supporting  activities  (e.g., 
human  resources)  to  play  their  proper  roles 

Although  not  all  of  these  methods  are  conventionally  associated  with  instructional 
strategy,  they  do  affect  the  quality  of  self-instruction  through  the  availability  of  (a)  quality 
courseware;  (b)  instructors  and  other  support  to  interact  with  distributed  students;  and  (c)  time 
set  aside  specifically  for  online  learning. 

Examination  of  the  methods  used  to  convey  course  content  indicated  rough  equivalence 
between  the  resident  and  blended  SCCC.  First,  segments  of  the  curriculum  that  required  group 
exercises  and  collaborative  learning  (e.g.,  staff  exercises,  network  management)  were  conducted 
as  resident  phases  in  blended-SCCC,  with  individual  leader  and  technical  skills  (e.g.,  staff 
knowledge,  signal  theory)  reserved  for  online  instruction.  Online  instruction  largely  was 
delivered  via  narrated  PowerPoint  lectures  and  interactive  courseware,  including  some 
equipment  simulation,  reflecting  the  appropriate  media  selections. 
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Despite  the  application  of  these  methods,  significant  differences  in  learning  experience 
did  exist,  and  were  related  to  both  the  instructorless  nature  of  the  DL  segments  of  the  course  and 
the  instructional  media  used.  One  noteworthy  difference  between  the  two  versions  of  the  SCCC 
was  the  degree  of  student  interaction  with  instructors  and  with  each  other  while  learning  the 
content  covered  in  the  online  phases  of  the  blended  SCCC  (Phases  II  and  IV).  Students  in  both 
focus  groups  unanimously  agreed  that  greater  interaction,  particularly  with  instructors,  during  the 
distributed  phases  of  instruction  would  be  desirable.  Specifically,  they  reported  being  more 
engaged  by  “authentic”  or  generative  assessment  activities  (e.g.,  creating  a  memorandum)  in 
which  they  produced  an  actual  leader  product  and  received  feedback  directly  from  the  instructor. 
They  did  not  feel  that  the  assessment  activities  in  the  interactive  courseware  were  useful  for  their 
learning  or  retention.  They  reported  that  these  assessments  were  too  easy  to  answer  correctly 
without  complete  knowledge  or  proficiency  with  the  content.  Students  also  reported  technical 
difficulties  with  scoring  interactive  courseware  assessments,  such  that  correct  answers  were 
counted  as  wrong  and  vice  versa. 

Students  also  desired  greater  interactivity  with  each  other  through  group  assignments  on 
leader  tasks  not  currently  covered  by  the  instruction.  They  did  not  wish  for  synchronous 
interaction,  citing  scheduling  challenges  associated  with  conducting  widely  distributed  group 
activities.  Nor  did  they  wish  for  longer  resident  phases.  Rather,  they  expressed  a  desire  for 
content  that  was  more  targeted  to  their  needs  as  leaders  and  managers  and  for  interaction  with 
others  as  appropriate  to  facilitate  such  learning.  The  collaborative  capabilities  of  the  software 
used  by  the  Fort  Gordon  LLC  to  deliver  curriculum  materials  (i.e.,  Blackboard)  permits 
discussion  and  community  development  among  distributed  learners,  although  it  is  not  currently 
used  in  this  way.  Proactively  leveraging  collaborative  capabilities  may  enhance  the  social  aspects 
of  distance  learning,  provide  a  straightforward  way  for  students  to  find  and  help  each  other,  and 
may  increase  the  number  of  SCCC-blended  students  who  actively  engage  in  course  materials 
through  interaction  with  the  instructor. 

Another  important  difference  observed  between  the  learning  experiences  of  resident-  and 
blended-SCCC  students  was  tied  to  the  methods  noted  by  Leonard,  Winkler,  Hove,  et  al.  (2001) 
that  relate  to  providing  support  and  incentives  to  enable  online  instruction.  Specifically,  students 
reported  not  being  compensated  for  the  time  they  spent  completing  online  coursework,  nor  were 
they  given  set-aside  training  time  by  their  commanders  or  employers  to  focus  on  education.  As  a 
result,  students  completed  the  minimum  necessary  amount  of  course  content  as  quickly  as 
possible  when  they  could  work  it  around  other  demands.  The  methods  they  reported  using 
included:  staying  up  late,  taking  vacation  or  sick  days  off  of  work,  putting  off  instruction  until 
the  last  minute  and  cramming,  and  simultaneously  doing  coursework  and  recreational  activities 
(e.g.,  watching  TV).  Focus  group  interviewees  unanimously  felt  that  non-resident  instruction  was 
beneficial  but  that  the  way  it  had  been  implemented  placed  a  significant  burden  on  the  learner. 
Although  it  was  not  explicitly  assessed  as  part  of  this  research  effort,  it  is  likely  that  such  a  belief 
would  not  enhance  affective  organizational  commitment,  and  could  possibly  have  a  negative 
impact. 


Similarly,  instructors  did  not  receive  assistance  or  time  set  aside  for  actively  facilitating 
online  learning,  which  can  be  very  time  consuming  but  essential  for  effective  distributed 
instruction  (Abell,  2000;  Hiltz,  1998;  Kreijns,  Kirschner,  Jochems,  2003).  The  current  design  of 
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most  of  the  online  courseware  used  in  blended-SCCC  obviates  the  instructor  except  as  a  course 
administrator.  Although  it  is  recognized  that  instructorless  education  would  save  money,  there  is 
a  potential  cost-benefit  tradeoff  that  is  worth  exploring. 

Equivalence  of  Learning  Performance.  The  revealed  differences  in  learning  experience 
raise  questions  about  the  equivalence  of  learning  performance  across  resident-  and  blended- 
SCCC  students.  As  described  previously,  both  resident-  and  blended-SCCC  capstone  staff 
exercises  were  observed  in  order  to  investigate  learning  performance.  This  exercise  was  selected 
because  it  was  held  at  the  end  of  the  course,  thus  increasing  the  likelihood  that  any  observed 
differences  would  be  due  to  prior  instruction.  In  both  the  resident  and  blended  versions  of  the 
SCCC,  the  combined  arms  staff  exercise  follows  training  in  staff  knowledge  and  requires 
students  to  play  staff  roles  with  which  they  may  be  unfamiliar.  The  mission  analysis  and  course 
of  action  development  phases  were  observed,  along  with  their  associated  briefings  to  the 
instructor. 

Some  differences  in  how  the  capstone  exercises  were  conducted  for  resident-  and 
blended-SCCC  students  should  be  described.  First,  blended  students  were  allocated  one  less  day 
to  complete  the  exercise.  These  students  made  up  for  the  missing  day  by  working  longer  hours 
on  the  days  they  conducted  the  exercise.  The  result  was  that  resident  students  worked  for  five, 
approximately  8-hour  days  and  the  blended  students  worked  for  four,  approximately  1 0-hour 
(and  sometimes  longer)  days.  Second,  the  instructor  was  much  more  involved  in  guiding  the 
exercise  and  providing  mentorship  with  blended  students  than  with  resident  students.  Third,  the 
two  blended  student  groups  observed  (N=7,N=%)  were  slightly  smaller  than  the  resident 
student  group  (N  =  9)  and  the  resident  group  had  a  foreign  national  student  from  Azerbaijan,  the 
country  serving  as  the  area  of  operations  for  the  exercise.  Fourth,  the  groups  had  differing 
numbers  of  members  who  had  served  on  a  battalion  staff  (N  =  4  and  2  for  the  blended  groups  and 
N  =  1  for  the  resident  group).  Fifth,  and  finally,  the  groups  differed  in  how  seriously  they  took 
the  exercise.  The  resident  group  was  much  more  relaxed  and  unconcerned  about  their 
performance  than  either  of  the  two  blended  groups  to  whom  it  meant  a  great  deal  to  perform 
well. 


Table  7  below  shows  examples  of  the  behaviors  sampled  during  classroom  observations 
and  the  assignment  of  a  level  of  cognitive  development.  As  shown  in  the  table,  the  performance 
of  both  groups  largely  reflected  reliance  on  rules  (e.g.,  doctrine,  the  operations  order),  variability 
in  analytical  capability,  and  focus  on  key  mission  factors  somewhat  in  isolation.  According  to 
Phillips,  Ross,  and  Shadrick  (2006),  these  behaviors  reflect  a  “Novice”  level  of  cognitive 
development.  A  Novice  level  of  cognitive  development  is  to  be  expected  from  the  students  at  this 
stage  in  their  careers  given  very  limited  experience  with  the  military  decision  making  process 
and  combined  arms  planning.  Students  at  this  stage  had  not  yet  acquired  the  knowledge  that 
enables  more  analytical  and  flexible  thinking,  and  were  asked  to  play  roles  that  they  will  not  play 
in  the  operational  environment. 

One  exception  to  this  observation  was  course  of  action  development  as  conducted  by  the 
resident  students.  In  this  phase  of  the  exercise,  the  resident  students  demonstrated  some  ability  to 
recognize  meaningful  elements  in  the  information  provided  and  specify  their  implications, 
particularly  for  enemy  action.  They  also  went  beyond  established  rules  to  address  unknowns 
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about  enemy  strength.  For  this  reason,  their  performance  in  this  phase  was  assigned  an 
“Advanced  Beginner”  rating. 


Table  7.  Capstone  Exercise  Performance  of  Resident-  and  Blended-SCCC  Student  Groups 


Student 

Group 

Exercise  Phase 

Example  Behaviors 

Level  of  Cognitive 
Development 

Resident 

Mission  Analysis 
(MA) 

-  High-level  descriptions  of  geography,  population, 

and  political/economic  situation  (i.e.,  regurgitation 
of  operations  order) 

-  Limited  enemy  analysis  (i.e.,  goals  not  explicitly 
stated) 

-  Battlefield  effects  analysis  identified  most  key 
implications  of  weather  and  terrain 

-  Facts  and  assumptions  moderately  well  identified 
and  articulated 

-  Implied,  specified,  and  essential  tasks  reflected 
some  conflation  of  the  mission  and  essential  tasks, 
some  overlap  in  specified  tasks  among  warfighting 
functions,  recognition  of  constraints  satisfactory 

-  Some  assumptions  reflected  lack  of  knowledge 

Novice 

Course  of  Action 
Development  (CO AD) 

-  Implications  of  enemy  courses  of  action  well 

articulated 

-  Good  analysis  of  enemy  patterns 

-  Enemy  adaptations  to  friendly  strengths  seen  as 
“weaknesses”  instead  of  thinking  enemy  tactics 

-  Effective  analysis  of  differing  tactics  among 
differing  threat  groups  and  most  likely  and 
dangerous  COA 

-  Modified  force  ratio  matrix  to  address  unknowns 

-  Evaluation  criteria  with  arbitrary  weights  that 
couldn’t  be  justified 

Advanced  Beginner 

Blended 

Mission  Analysis 
(MA) 

-  Significant  amount  of  time  (1+  hrs)  spent  reading 

doctrinal  manuals  prior  to  beginning  analysis 

-  Some  conflation  of  facts  and  tasks 

-  Instructor  provided  guidance  on  considering  all 
assets,  information  requirements,  and  facts  vs. 
assumptions 

-  Slide  organization  deviates  from  normal  procedure, 
presentation  somewhat  confusing 

-  Commander’s  Critical  Information  Requirements 
cut  and  pasted  from  OPORD 

-  One  group  to  do  significant  revisions  of  MA  brief 

-  Restated  mission  one  level  of  analysis  too  high 

Novice 

Course  of  Action 
Development  (COAD) 

-  Instructor  advised  students  not  to  exert  command 

and  control  too  many  levels  down 

-  Conflation  of  the  mission,  purpose,  and  tasks  to  be 
accomplished  with  the  CO  As 

-  Levels  of  analysis  at  the  platoon  level  (i.e.,  too 
low) 

-  Good  evaluation  criteria 

Novice 

The  data  in  Table  7  suggest  that  there  was  little  practical  difference  in  the  performance  of 
blended-  and  resident-SCCC  students.  Students  achieved  similar  levels  of  performance,  although 
blended  student  groups  received  a  great  deal  more  mentoring  and  guidance  from  the  instructor  to 
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achieve  the  level  of  performance  they  did.  One  of  the  blended  groups  had  to  redo  significant 
portions  of  their  mission  analysis  before  completing  their  brief  to  the  instructor. 

Recommendations 

Solutions  for  building  interpersonal  interactivity  into  online  courseware  generally  are  not 
simple,  nor  are  they  inexpensive.  Instructor  time  is  required  for  collaboration  and  additional 
course  developer  time  (and  expertise)  is  required  to  build  computer-based  training  that  has 
interactivity  as  its  key  mode  for  delivering  instruction  and  conducting  assessment.  Blackboard 
currently  supports  interactivity  among  instructors  and  students,  but  procedures  must  be  put  into 
place  and  faculty  developed  in  order  to  leverage  these  capabilities.  In  the  case  of  leader 
education,  as  in  the  case  of  MOSQ  instruction,  the  development  of  instructors,  course 
developers,  and  the  overall  technology-assisted  instruction  process  can  only  be  accomplished 
through  close  coordination  and  well-defined  roles  among  resident  instruction  managers  and  the 
LLC  staff.  In  the  MANSCEN  LLC,  courseware  development  is  a  critical  function  of  the  LLC 
staff,  a  model  potentially  worth  exploring  for  the  Fort  Gordon  LLC. 

Assessment  of  On-Demand  Training 
The  Sample  -  Mobile  Training  Teams  and  Unit  Universities 

As  stated  previously  in  this  report,  much  of  what  the  Fort  Gordon  LLC  offers  to  lifelong 
learners  may  be  considered  training  on  demand.  Simulation  downloads,  online  courseware,  and 
discussion  forums  all  enable  24/7  access  to  learning  content  at  the  initiative  of  the  learner. 
However,  in  order  to  clarify  the  presentation  of  this  report,  these  aspects  of  the  Fort  Gordon  LLC 
have  been  considered  separately.  Mobile  training  teams  and  Unit  Universities  represent  outputs 
uniquely  produced  by  the  Fort  Gordon  LLC,  whereas  simulations,  blended  learning,  and 
knowledge  management  have  been  implemented  extensively  throughout  the  Army.  For  this 
reason,  mobile  training  teams  and  Unit  Universities  require  their  own,  novel  assessment  criteria 
and  focused  analysis. 

The  critical  function  served  by  On-Demand  Training  is  the  provision  of  MOS 
sustainment  training  independently  of  the  institutional  education  cycle.  The  purpose  of  On- 
Demand  Training  is  to  support  the  Army’s  apprenticeship  model  of  instruction,  enabling 
pedagogically  sound  and  technologically  advanced  on-the-job  training.  It  is  possible  for  On- 
Demand  Training  to  improve  the  cost-effectiveness  of  training,  but  it  is  more  important  for 
enhancing  mission  readiness  because  it  serves  as  an  alternative  to  institutional  modes  of 
instruction.  Enhanced  mission  readiness  stems  from  the  accelerated  skill  development  and 
increased  retention  enabled  by  just-in-time  instruction. 

Assessment  Questions  &  Method 

The  present  research  assessed  the  output  and  outcomes  associated  with  the  use  of  On- 
Demand  Training  for  enhancing  proponent  outreach.  Specifically,  the  following  questions  were 
asked  and  associated  metrics  used  to  focus  data  collection: 
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o  Question:  Is  On-Demand  Training  being  adopted? 

o  Metric:  %  of  target  audience  registered  to  use  the  system 
o  Question:  Does  On-Demand  Training  enable  just-in-time  learning? 

o  Metric:  %  of  registered  users  who  state  that  training  received  was  timely  and 
could  not  have  been  possible  any  other  way 
o  Question:  Is  On-Demand  Training  cost-effective? 

o  Metric:  Reduction  in  cost  to  conduct  mobile  training  teams  before  and  after  LLC 
implementation 

These  questions  were  answered  through  a  combination  of  interviews  and  review  of 
archival  data.  Interviews  were  conducted  with  the  Fort  Gordon  LLC  Extension  Campus 
Coordinator,  the  LLC  Program  Manager,  the  Signal  Center  Chief  of  Resident  Training 
Management,  the  Chief  of  the  Training  Development  Cell  located  at  the  Signal  Center  NCO 
Academy,  and  ten  training  supervisors  who  established  or  oversaw  Unit  Universities  with  the 
Fort  Gordon  LLC.  Interviews  with  training  supervisors  were  conducted  in  order  to  capture 
concrete  information  about  how  the  Unit  Universities  enabled  just-in-time  training  and  enhanced 
readiness.  Archival  data  reviewed  included  cost-effectiveness  analyses  conducted  by  the  Fort 
Gordon  LLC  staff  and  mobile  training  team  cost  data  provided  by  the  Signal  Center  Chief  of 
Resident  Training  Management. 

Interviews  conducted  with  Unit  University  training  supervisors  represent  a  relatively 
modest  sample  due  to  several  factors.  First,  three  Unit  Universities  were  used  to  conduct  non¬ 
resident  25B10  instruction  (i.e.,  they  supported  the  High-Tech  Regional  Centers  at  Tobyhanna 
and  Sacramento  and  the  Professional  Education  Center  in  Little  Rock),  which  was  discussed 
previously  in  this  report.  These  Unit  Universities  were  not  considered  part  of  On-Demand 
Training,  bringing  the  sample  number  down  to  44.  Second,  of  the  seven  original  Unit 
Universities  (up  and  running  by  November  2006),  two  of  the  associated  units  had  been 
deactivated.  Third,  of  the  remaining  42  Unit  Universities,  18  training  supervisors  were  contacted 
and  eight  did  not  respond  to  a  request  for  an  interview.  Fourth  and  finally,  of  the  ten  training 
supervisors  contacted,  five  indicated  that  their  Unit  Universities  were  too  recently  established  to 
have  much  to  say  about  impact.  Because  the  remaining  24  (42-18)  Unit  Universities  had  also 
been  very  recently  established,  further  attempts  to  contact  training  supervisors  were  suspended. 

Summary  of  Findings 

Unit  trainers  interviewed  unanimously  and  enthusiastically  endorsed  the  importance  of 
Unit  Universities  to  enhancing  their  units’  mission  readiness.  The  LLC  staff  has  launched  as 
many  Unit  Universities  as  personnel  and  hardware  limitations  would  allow,  with  recent  upgrades 
to  hardware  poised  to  accelerate  the  process  in  the  immediate  future.  Most  of  the  unit  trainers 
interviewed  had  newly  established  Unit  Universities,  but  easily  cited  the  ways  in  which  they 
thought  training  on  demand  would  provide  unique  and  cost-effective  opportunities  for  refresher 
and  new  equipment  training.  Mobile  training  teams  supported  by  the  LLC  technologies  enabled 
at  least  partial  cost  reduction  to  the  Signal  Center’s  ongoing  efforts  to  reach  out  to  operational 
units  in  need  of  pre-  and  post-deployment  training.  The  delivery  of  training  to  a  deployed  unit 
via  a  virtual  mobile  training  team  represented  a  unique  mode  of  proponent  outreach  that  made 
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cost-effective  use  of  existing  LLC  personnel  and  schoolhouse  experts  through  close 
coordination. 

Kinney  (2005)  presented  some  external  factors  that  unit  trainers  cited  as  important  for 
enabling  lifelong  learning  in  units.  Among  these  factors  were  time  to  train,  command  emphasis, 
and  incentives  to  train  during  down  time.  All  three  of  these  factors  also  were  cited  by 
interviewees  as  currently  limiting  the  impact  of  Unit  Universities.  Other  limiting  factors  cited  by 
unit  trainers  were  lack  of  awareness  of  Unit  Universities  and  the  difficulty  associated  with 
acquiring  course  content  that  is  not  readily  available  online.  Although  the  first  three  factors  are 
outside  the  sphere  of  influence  of  the  LLC  and  the  Signal  Center,  these  additional  factors  are  not. 
They  could  be  addressed  through  closer  coordination  among  content  developers  and  providers 
and  increased  marketing  outreach  of  the  lifelong  learning  concept. 

Detailed  Findings 

Adoption  of  On-Demand  Training.  At  the  time  this  analysis  was  conducted,  the  Fort 
Gordon  LLC  hosted  47  Unit  Universities.  The  Extension  Campus  Coordinator  estimated  that  1 80 
Unit  Universities  would  be  necessary  to  reach  every  unit  in  the  Signal  Corps  and  that  more  than 
600  would  be  necessary  to  reach  signal  and  non-signal  units  who  could  benefit  from  reach  back 
to  the  Signal  Center.  Forty-seven  Unit  Universities  represents  approximately  30%  of  the  Signal 
target  audience  and  8%  of  the  Army-wide  target  audience,  a  modest  percentage  that  has  grown 
significantly  in  the  past  year  (1%  to  8%).  According  to  the  LLC  staff  interviewed,  constraints  on 
the  growth  of  the  Unit  University  initiative  have  been  due  largely  to  limited  server  space  to  host 
Unit  Universities  (which  produced  concerns  about  backup  activities)  and  lack  of  staff  time 
available  to  get  the  word  out  about  On-Demand  Training.  Both  LLC  staff  members  interviewed 
reported  a  surge  in  interest  in  Unit  Universities  following  visits  by  the  former  Deputy  Chief  of 
Signal  to  bring  word  of  LandWarNet  eSignal  to  operational  units.  Recent  acquisitions  of  new 
hardware  should  enable  the  hosting  of  more  Unit  Universities.  If  the  present  rate  of  growth  (i.e., 
roughly  six  times  the  number  of  Unit  Universities  in  one  year  as  were  present  the  year  previous) 
continues,  the  Army-wide  target  audience  should  be  reached  within  approximately  2  years.  A 
more  realistic  estimate,  given  current  limitations  in  personnel  and  courseware  availability  would 
probably  be  about  10  years. 

It  is  difficult  to  estimate  an  appropriate  adoption  rate  for  mobile  training  teams,  given  the 
spontaneous  nature  of  requests  for  On-Demand  Training  and  the  potentially  very  broad  target 
audience.  For  live  mobile  training  teams,  LLC  capabilities  are  used  more  as  a  facilitative 
capability  (i.e.,  to  save  printing  and  reproduction  costs)  than  as  an  enabling  capability  (i.e., 
making  it  possible  to  send  out  a  mobile  training  team).  The  Signal  Center  Chief  of  Resident 
Training  Management  stated  that  it  can  be  most  cost-effective  to  conduct  training  at  the 
schoolhouse,  so  emphasis  is  placed  on  maximizing  the  ability  of  units  to  come  to  Fort  Gordon. 
For  this  reason,  effective  use  of  the  Fort  Gordon  LLC  to  deliver  blended  learning,  among  other 
services,  may  actually  help  reduce  the  frequency  of  mobile  training  team  deployment. 

One  way  to  estimate  the  adoption  rate  for  virtual  mobile  training  teams  is  to  compare  the 
frequency  of  usage  compared  to  live  mobile  training  teams.  The  Chief  of  Resident  Training 
Management  reported  that  since  October  2005,  the  Signal  Center  sent  close  to  20  live  mobile 
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training  teams  per  year.  Since  its  inception  in  2002,  Fort  Gordon  has  deployed  one  virtual  mobile 
training  team.  As  with  Unit  Universities,  staffing  shortfalls  and  limited  marketing  activity  appear 
to  have  contributed  to  modest  adoption  rate.  It  is  not  widely  known  that  the  Fort  Gordon  LLC 
can  provide  virtual  mobile  training  teams.  Even  so,  significant,  close  coordination  among  LLC 
staff  and  Signal  Center  instructors  and  course  developers  would  be  necessary  to  prepare  the 
appropriate  training  content. 

Just-in-time  training.  Of  the  ten  Unit  University  training  supervisors  interviewed,  five 
indicated  that  their  Unit  University  was  too  newly  established  to  yet  have  an  idea  of  its  impact. 
Nine  of  the  10  interviewees  were  very  enthusiastic  about  the  possibilities  for  their  Unit 
University  to  enable  MOS  sustainment  training  and  reach  back  to  the  schoolhouse. 1 1  These 
interviewees  reported  that  they  established  a  Unit  University  to  enhance  mission  readiness 
through  (1)  training  on  new  equipment  assigned  to  the  unit;  (2)  training  on  MOS-specific  skills 
that  were  needed  for  deployment,  but  that  could  not  be  trained  feasibly  in  the  schoolhouse  (e.g., 
not  enough  seat  reservations);  (3)  increased  confidence  enabled  by  reach  back  to  Signal  Center 
resources;  (4)  reinforcement  of  existing  theory  or  skills;  (5)  reduced  time  away  from  the  unit  for 
training;  and  (6)  reduced  training  cost. 

It  is  important  to  note  that  the  motivation  to  establish  Unit  Universities  appeared  to  have 
come  from  a  need  to  enhance  readiness  more  directly  than  by  improving  personnel  or  training 
percentages  as  stated  in  a  unit  readiness  report.  Improving,  for  example,  a  personnel  percentage 
requires  that  the  number  of  duty-MOS-qualified  Soldiers  in  a  unit  changes.  Such  a  change 
requires  institutional  education  conducted  either  entirely  at  the  schoolhouse  or  provided  as  a 
blended  learning  solution  or  mobile  training  team.  The  Unit  Universities  of  nine  of  the  ten 
training  supervisors  interviewed  were  established  rather  as  workarounds  to  the  challenges  of 
going  through  the  formal  educational  system  in  a  timely  fashion.  The  nature  of  how  readiness  is 
enhanced  by  On-Demand  Training  further  highlights  the  unique  characteristics  of  this  initiative 
and  its  ability  to  provide  training  that  could  not  be  conducted  in  any  other  way. 

That  said,  the  established  Unit  Universities  have  not  yet  achieved  optimum  effectiveness 
in  enhancing  just-in-time  competency,  as  reported  by  the  training  supervisors  interviewed.  The 
primary  barriers  to  effectiveness  that  they  listed  were  (1)  limited  time  to  conduct  on-the-job 
training;  (2)  lack  of  incentives  for  conducting  additional  training  outside  of  daily  duties;  (3)  lack 
of  command  emphasis  on  using  digital  training  for  augmenting  work  performance;  and  (4) 
challenges  getting  schoolhouse  courseware  released  online  or  getting  special  purpose  courseware 
created.  The  majority  of  these  barriers  are  beyond  the  direct  sphere  of  influence  of  the  LLC, 
although  initiating  and  maintaining  the  development  of  social  networks  among  itself,  unit 
commanders,  and  schoolhouse  education  providers  will  be  critical  for  facilitating  the  delivery  of 
training  on  demand  and  ensuring  it  will  be  used. 

Cost-ejfectiveness.  The  total  cost  of  sending  mobile  training  teams  (MTTs)  varies  a  great 
deal,  depending  on  the  location  where  the  team  is  to  be  sent  (costs  include  travel  and  possibly 


1 1  One  of  the  ten  interviewees  was  using  his  Unit  University  to  deliver  the  non-resident  Sergeants  Major  course.  He 
was  also  very  enthusiastic  about  the  capabilities  of  his  Unit  University,  but  from  the  stance  of  providing  institutional 
instruction  rather  than  just-in-time  training.  Fifteen  percent  (7/47)  of  Unit  Universities  are  used  to  support  blended 
learning  for  institutional  education. 
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hazardous  duty  pay),  the  duration  of  the  training  (instructor  costs),  the  number  of  students 
(instructor  and  printing  and  reproduction  requirements),  and  equipment  requirements.  The  Fort 
Gordon  LLC  can  assist  with  reducing  these  costs  by  making  course  content  available  online, 
thereby  reducing  the  printing  and  reproduction  costs  of  conventional  MTTs,  or  by  providing 
training  virtually  (i.e.,  virtual  MTTs).  Both  types  of  MTT  have  been  deployed  by  the  Fort 
Gordon  LLC. 

MTTs  with  training  content  (e.g.,  lecture  slides)  hosted  in  an  LLC  represent  relatively  a 
minor  reduction  in  cost,  printing  and  reproduction  only.  In  this  case,  instructors  still  must  be  sent 
to  the  training  location,  as  (potentially)  does  the  equipment.  The  feasibility  of  sending  a  mobile 
training  team  therefore  is  a  function  of  the  number  of  students  who  need  the  training  and  the 
readiness  cost  (i.e.,  time  away  from  unit)  associated  with  sending  these  students  to  the 
schoolhouse,  where  equipment  and  instructors  are  already  located,  or  deploying  these  personnel 
without  the  training.  One  conventional  MTT  enabled  by  the  Fort  Gordon  LLC  was  sent  to  Fort 
Hood,  and  one  to  be  sent  back  to  the  same  location  was  determined  to  be  infeasible  given  the 
relatively  small  number  of  students.  Factors  therefore  somewhat  beyond  the  sphere  of  influence 
of  the  LLC  (i.e.,  availability  of  equipment  at  the  unit  to  be  trained  and  instructor  costs)  ultimately 
determined  the  deployability  of  a  conventional  MTT. 

Since  its  launch  in  2002,  the  Fort  Gordon  LLC  has  deployed  one  virtual  mobile  training 
team.  As  presented  in  the  Fort  Gordon  LLC  VIP  brief,  a  unit  in  Iraq  discovered  that  it  had  to 
work  with  a  piece  of  satellite  communications  equipment  with  which  it  was  completely 
unfamiliar.  The  unit  requested  training  on  this  equipment  in  order  to  successfully  move,  operate, 
and  maintain  the  equipment.  In  two  weeks’  time,  the  LLC  staff,  together  with  instructors  at  the 
Signal  Center,  developed  videotaped  demonstrations  of  how  to  perform  the  required  functions 
and  hosted  it  online  through  the  LLC  along  with  operation  manuals,  maintenance  manuals,  and 
instructor  slides  and  notes.  The  estimated  cost  savings  associated  with  conducting  the  training 
virtually  was  more  than  $400,000. 

Present  examination  of  archival  cost  estimates  for  sending  a  MTT  to  Iraq  indicated  that 
the  vast  majority  of  the  costs  incurred  by  such  a  team  would  be  personnel  salary  and  travel  costs. 
Blended  learning  solutions  that  have  been  piloted  to  conduct  25B10  MOSQ  instruction  with  the 
National  Guard  demonstrates  an  option  for  decreasing  the  actual  costs  of  deploying  virtual 
mobile  training  teams  by  reducing  the  duration  of  the  team’s  deployment.  In  any  case,  a  cost 
approaching  $400,000  to  send  a  mobile  training  team  to  a  hazardous  area  such  as  Iraq,  raised  the 
question  of  how  often  this  occurs  and  therefore  how  many  opportunities  an  LLC  has  to  reduce 
costs  by  providing  virtual  teams  instead  of  live  ones.  The  Fort  Gordon  LLC  has  not  had  the 
opportunity  to  provide  another  virtual  mobile  training  team  (instead  of  a  live  mobile  training 
team)  in  the  six  years  it  has  been  established.  This  reality  highlights  the  importance  of  LLCs  and 
virtual  mobile  training  teams  to  making  training  possible,  perhaps  more  so  than  saving  money  to 
conduct  training. 

Recommendations 

The  impact  of  On-Demand  Training  on  mission  readiness  and  cost  savings  is  a  direct 
function  of  the  adoption  of  the  Fort  Gordon  LLC  by  units  in  the  field.  Increased  adoption  of  Unit 
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Universities  and  virtual  mobile  training  teams  will  improve  the  likelihood  that  skills  can  be 
acquired  or  refreshed  just  in  time  and  enhances  the  effectiveness  (both  in  terms  of  pedagogy  and 
resource  requirements)  of  proponent  outreach  to  lifelong  learners.  The  rate  of  expansion  of  Unit 
Universities  suggests  that  the  adoption  of  On-Demand  Training  is  accelerating,  but  also  that 
there  are  opportunities  for  the  Fort  Gordon  LLC  and  Signal  Center  to  extend  their  outreach 
through  increased  marketing  activities. 

In  order  to  fully  understand  the  readiness  impact  of  Unit  Universities,  some  follow-up 
research  is  necessary.  This  research  should  be  facilitated  by  the  LLC  staff  through  working 
documentation  of  the  following  information: 

o  A  history  for  each  Unit  University  (i.e.,  how/why  it  was  established,  how  long  it  took, 
how  the  requester  heard  about  LandWarNet,  etc.) 

o  The  training  content  hosted  in  each  Unit  University 

o  The  history  of  the  training  content  (i.e.,  whether  it  was  developed  specifically  for  the  Unit 
University  or  collected  from  an  existing  source,  the  social  networks  involved  in  making 
the  content  available,  etc.) 

In  addition,  it  would  be  beneficial  if  the  LLC  staff  conducted  periodic  follow-up  phone 
interviews  with  the  training  supervisors  in  charge  of  the  Unit  Universities  so  that  stories  of  how 
the  system  is  working  and  what  the  challenge  areas  are  can  be  collected  and  acted  upon. 
Documentation  of  the  external  factors  that  limit  the  impact  of  Unit  Universities  may  support 
efforts  to  exact  change  in  other  areas  that  could  make  training  more  adaptive  and  flexible  in  the 
future. 


LLC  SELF-ASSESSMENT 

LLC  self-assessment  enables  TRADOC  to  leverage  the  findings  of  this  ARI  program  of 
research  in  order  to  develop  the  lifelong  learning  initiative  well  into  the  future.  For  this  reason, 
effort  was  devoted  to  exploring  the  requirements  for  conducting  LLC  self-assessment  using  the 
revised  assessment  framework  and  analyzing  the  feasibility  of  these  requirements.  The  goal  was 
to  develop  a  plan  for  conducting  LLC  self-assessment  that  could  be  readily  adopted  and  also 
produce  meaningful  data  about  learning  effectiveness  and  organizational  impact.  Assessment 
lessons  learned  throughout  the  research  program  were  collected  to  assist  in  forming  this  plan  and 
are  summarized  below. 


LLC  Assessment  Lessons  Learned 

Lesson  #1  -  The  Rate  of  Change  in  LLC  Activities  and  Outputs  is  Very  High 

As  described  previously,  interpreting  the  results  of  LLC  assessments  must  take  into 
account  the  fact  that  LLCs  change  rapidly  and  often.  Among  the  LLC  characteristics  that  change 
are: 


o  Terminology:  The  use  of  multiple  terms  to  refer  to  the  same  things  (or  vice  versa,  to  use 
the  same  term  to  refer  to  multiple  things)  is  widespread.  For  example,  the  Fort  Gordon 
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LLC  also  is  known  to  various  users  as  the  University  of  Information  Technology, 
LandWarNet  e-Signal  and  LandWarNet  eUniversity.  Other  multi-use  terminology 
includes  referents  to  modes  of  delivering  instruction  (e.g.,  distributed  vs.  distance 
learning  vs.  blended  learning  and  simulation  vs.  simulator  vs.  courseware),  which  often 
are  used  independently  of  their  definitions. 

o  Courses  offered:  LLCs  are  characterized  by  significant,  responsive  growth  to  training  and 
education  requirements  that  emerge  from  both  proponent  schoolhouses  and  the  field.  The 
result  of  this  adaptivity  is  the  emergence  of  different  types  of  courses  which  are  best 
delivered  using  different  types  of  instructional  strategy,  educational  technology,  and,  of 
course,  different  content.  The  LLCs  adapt  in  response,  adding  new  components, 
modifying  interface  layouts,  and  changing  technical  capabilities, 
o  Support  infrastructure:  Since  the  Signal  Center’s  University  of  Information  Technology 
was  established  as  the  first  LLC  and  the  initial  executive  agent  for  the  lifelong  learning 
initiative,  several  changes  have  been  made  regarding  where  strategic  planning  and 
decision  making  are  executed.  The  location  and  means  for  providing  server  support  and 
technical  assistance  also  have  undergone  change,  with  plans  for  further  evolution, 
o  Use  practices:  The  use  of  LLCs  (and,  by  extension,  interface  their  layouts)  evolves  as  the 
technical  staff,  instructors,  course  developers  and  other  stakeholders  develop  processes 
for  working  together  and  enhance  their  ability  to  leverage  the  technologies  involved. 
Other  driving  forces  behind  change  in  use  practices  involves  the  integration  of  additional 
stakeholders  who  enhance  LLC  functioning,  such  as  faculty  developers,  quality  assurance 
professionals,  marketing  personnel,  course  administrators,  and  so  on. 
o  Personnel:  LLCs  are  run  by  contractor  personnel,  which  change  at  least  on  a  periodic 
basis  when  a  challenger  successfully  wins  a  re-compete.  Changes  in  contractors  generally 
do  not  involve  a  complete  turnover  in  personnel,  but  what  change  in  personnel  does  occur 
results  in  process  and  data  loss  as  new  hires  are  brought  up  to  speed, 
o  Assessment  goals:  As  LLCs  evolve,  the  reason  for  assessing  them  changes,  as  does  the 
target  of  assessment.  Assessment  that  was  initially  conducted  to  justify  the  existence  of  a 
particular  LLC  (or  of  the  broader  LLC  concept)  may  later  be  conducted  to  evaluate  the 
impact  of  introducing  specific  technologies,  faculty  development  interventions,  support 
infrastructure  modifications,  or  other  initiatives.  As  LLCs  become  broader  in  scope,  the 
assessment  spotlight  may  shift  across  components  because  an  assessment  of  all 
components  simultaneously  would  be  too  labor  intensive, 
o  Technology:  The  technology  used  as  the  backbone  of  LLCs  is  always  subject  to  change. 
Existing  technologies  may  be  substituted  by  more  cost-effective  or  powerful  alternatives. 
New  technologies  may  be  added  to  leverage  advances  in  capability  or  to  enable  the 
achievement  of  newly  identified  learning  objectives.  Individual  proponents  may  augment 
the  LLC  foundational  technology  with  applications  of  their  own  in  order  to  meet  the 
particular  needs  of  their  learners. 

Lesson  #2  -  The  Assessment  Methods  Necessary  to  Capture  Meaningful  Data  Require  a  “ Human 
in  the  Loop  ” 

A  range  of  methods  may  be  used  to  capture  assessment  metrics,  including  system 
analysis  (e.g.,  content  analysis  of  discussion  forums),  surveys,  interviews,  focus  groups,  archival 
data  analysis  (e.g.,  financial  data,  programs  of  instruction,  course  grades,  etc.),  and  classroom 
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observation.  Unfortunately,  none  of  these  methods  are  suited  for  full  automation  for  the 
following  reasons: 

1 .  Although  the  LLC  Assessment  Framework  identifies  metrics  that  will  provide 
information  about  what  data  to  collect,  a  person  is  needed  to  develop  the  measures  used 
to  capture  metric  data.  Reuse  of  such  measures  as  surveys,  interview  and  focus  group 
protocols,  observer  checklists,  content  analysis  rubrics,  etc.,  is  possible,  and 
recommended  as  a  means  of  partial  automation,  but  some  modification  will  be  necessary 
over  time  to  address  the  kinds  of  changes  described  above. 

2.  Some  data  may  be  collected  automatically  (e.g.,  survey  data),  but  most  methods  require  a 
person  to  gather  data  from  other  people  (e.g.,  interviews,  focus  groups,  classroom 
observations).  There  are  methods  for  partially  automating  this  process,  for  example, 
providing  means  for  users  to  submit  comments  and  feedback  instead  of  conducting 
interviews  or  using  grades  instead  of  classroom  observations.  The  tradeoff  associated 
with  partial  automation  is  between  information  volume  and  ease  of  administration. 
Generally  speaking,  the  easier  it  is  to  administer  a  measure,  the  less  information  it 
provides.  A  human  is  required  to  judge  the  cost-benefit  tradeoff  of  selecting  among 
assessment  methods. 

3.  Metric  data  come  from  numerous  sources,  requiring  the  participation  and  support  of 
people  both  within  and  outside  the  proponent  schoolhouse.  Automated  data  collection 
likely  would  require  the  integration  of  multiple  data  sources  including  not  only  databases 
(automated  and  human-manipulated)  but  also  paper-based  repositories.  A  person  would 
be  necessary  to  integrate  the  information  from  these  numerous  sources. 

4.  Once  metric  data  have  been  captured,  a  person  is  required  to  analyze  and  interpret  the 
data.  Survey  software  can  automatically  summarize  response  data,  for  example,  but  it 
cannot  identify  the  implications  of  these  data  for  the  functioning  of  the  LLC.  Similarly, 
activity  data  may  be  collected  automatically  by  LLC  technologies,  but  these  data  paint 
only  a  partial  picture  of  effectiveness  and  impact.  Several  data  inputs  must  be  analyzed 
and  integrated  to  fully  understand  LLC  functioning. 

5.  Interpretation  of  metric  data  also  needs  a  human  to  specify  the  caveats  to  generalizing 
and  acting  on  the  findings.  For  example,  a  person  must  be  involved  in  determining 
whether  a  representative  sample  was  used  for  classroom  observations  and,  when  the 
sample  is  not  representative,  what  limits  are  placed  on  where  or  when  the  findings  apply. 

6.  Finally,  a  person  is  required  to  communicate  the  assessment  results  to  others.  At  some 
point  in  the  distant  future,  dashboard  technologies  (requiring  human  data  entry  and/or 
significant  database  integration)  could  be  designed  to  support  the  flexible  and  broad 
reporting  of  assessment  results.  The  design  of  such  a  dashboard,  however,  requires  a 
more  stable  assessment  context  than  is  currently  the  case  for  LLCs.  Assessment  metrics 
must  be  extensively  validated  and  widely  accepted  before  serving  as  the  design  basis  for 
an  assessment  dashboard. 

Lesson  #3  -  Face  Time  is  Critical  for  Conducting  Assessment 

Face  time  is  critical  for  conducting  assessment  for  multiple  reasons.  First,  visits  to  the 
proponent  schoolhouse  and  interviews  with  key  stakeholders  are  required  to  fully  understand  the 
complexities  of  that  proponent’s  LLC.  Although  on  paper  LLCs  share  the  same  basic 
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technologies,  functions,  and  goals,  LLCs  differ  widely  in  their  histories  and  particular  areas  of 
emphasis.  Physically  being  present  at  the  schoolhouse  facilitates  contact  with  a  larger  and  more 
representative  sample  of  people  involved  in  running  the  LLC  and  providing  instruction. 
Stakeholders  overlooked  by  an  assessor  who  has  not  visited  the  proponent  could  potentially 
provide  useful  information  or  could  provide  access  to  additional  people  of  which  the  assessor 
was  unaware.  When  the  connections  among  stakeholders  and  other  people  with  information  arise 
during  site  visits,  it  is  a  simple  matter  to  capitalize  on  such  networks  by  setting  up  meetings  or 
dropping  by  an  office.  Personal  introductions  facilitated  by  stakeholders  increase  the  likelihood 
of  a  productive  exchange  with  people  unfamiliar  with  the  assessor. 

Second,  face-to-face  meetings  help  to  build  trust  on  the  part  of  stakeholders  and  other 
providers,  which,  in  an  evaluative  context,  may  not  come  easily.  Multiple  meetings  often  are 
required  to  gain  access  to  people  and  data  that  can  present  the  most  complete  picture  of  LLC 
functioning.  Periodic,  face-to-face  updates  to  stakeholders  and  providers  on  assessment  findings 
go  a  long  way  toward  ensuring  shared  understanding  of  how  the  program  is  doing  and  achieving 
mutual  satisfaction  with  how  results  are  presented.  Email,  phone  and,  to  some  degree, 
videoteleconference  contact  among  assessors,  stakeholders,  and  other  providers  reduce  the 
observability  of  visual  cues  of  discomfort  with  the  process  and  therefore  prevent  opportunities  to 
amend  one’s  approach.  In  addition,  physical  distance  between  these  people  makes  the  assessor 
seem  more  remote,  allowing  interpretations  of  the  assessor’s  motives  to  become  more  subject  to 
the  concerns  and  unverified  assumptions  of  the  people  whose  program  is  being  assessed. 

Lesson  # 4  -  Collecting  Readiness  Data  Requires  Extensive  Field  Work 

Aside  from  theoretical,  proxy  measures  of  readiness,  such  as  organizational  commitment 
(e.g.,  McGonigle,  Casper,  Meiman,  Cronin,  Cronin,  &  Harris,  2005),  there  are  two  ways  to 
collect  readiness  data.  First,  organizational-level  readiness  data,  such  as  personnel  and  training 
readiness  status,  may  be  collected  via  queries  of  various  Army  databases.  Directly  querying  these 
databases  necessitates  the  appropriate  authority  and  need  to  know,  which  typically  is  reserved  for 
investigators  from  within  the  Army  or  those  research  institutions  closely  affiliated  with  it  (e.g., 
RAND  Corporation,  Institute  for  Defense  Analyses).  Without  this  authority  and  affiliation, 
extensive  social  networks  are  required  to  collect  the  same  data  indirectly.  Such  social  networks 
must  involve  well-established  relationships  with  Army  personnel  and/or  large  contractors  who 
work  actively  with  the  necessary  databases  and  are  willing  to  support  the  investigation. 

Second,  the  collection  of  unit-level  readiness  data,  such  as  time  to  competency,  requires 
close  contact  with  unit  trainers  and  commanders  in  the  field  who  receive  graduates  and  who 
participate  actively  in  fostering  lifelong  learning  in  their  subordinates.  As  demonstrated  in  the 
present  research  effort,  it  is  possible  to  collect  such  readiness  data  via  phone  interviews  and 
surveys,  but  a  “boots-on-the-ground”  perspective  would  be  more  effective  in  collecting  plentiful 
and  detailed  information  about  how  engagement  in  lifelong  learning  can  be  translated  into  the 
savings  of  life,  time,  and  money  (particularly  in  a  deployed  setting).  Such  detail  would  enhance 
the  qualitative  causal  model  of  LLC  impact  and  lead  to  the  development  of  new  metrics  and 
measures.  Unfortunately,  such  detail  would  be  costly  and  time  consuming  to  gather,  in  part 
because  lifelong  learning  graduates  are  widely  distributed  and  difficult  to  track  individually. 
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Implications  for  Using  the  LLC  Assessment  Framework 
Lesson  #/  Implications 

The  rapid  rate  of  change  discovered  in  Lesson  #1  (Rapid  Change)  indicates  that  some 
modification  to  the  LLC  Assessment  Framework  will  be  required  to  conduct  LLC  self- 
assessments  in  the  future.  The  present  research  suggests  that  the  nature  of  these  modifications 
will  be  the  addition  or  modification  of  logic  model  component  elements,  as  well  as  new  or 
modified  metrics  and  measures,  provided  that  new  use  practices  or  technologies  (not  anticipated 
by  the  current  framework)  are  assessed.  Activities  identified  in  the  revised  LLC  Assessment 
Framework  likely  will  have  to  be  adjusted  somewhat  as  the  LLC  concept  itself  is  expanded  to 
regional  and  enterprise  level.  For  instance,  actions  on  the  part  of  regional  and  enterprise  LLC 
technical  staff  to  facilitate  collaboration  (distributed  and  face-to-face)  with  course  developers, 
instructors,  administrators,  and  subject  matter  experts  must  be  a  critical  target  for  future  LLC 
assessments. 

A  second  implication  of  Lesson  #  1  is  that  the  cost  of  automating  LLC  assessment  likely 
would  significantly  outweigh  the  benefit.  Oversight  would  be  required  to  ensure  that  automated 
assessment  measures  use  the  correct  terminology,  address  the  appropriate  technologies,  apply  to 
the  right  courses,  and  so  on,  which  will  change  often  over  time.  Similarly,  administration  would 
be  necessary  in  order  to  verify  the  accuracy  and  quality  of  data  collected  automatically  in  the 
context  of  frequent,  rapid  change.  The  actions  that  assessment  personnel  would  have  to  take  to 
exert  the  proper  control  over  data  flow  would  amount  to  functionally  the  same  activities  that 
would  be  conducted  to  assess  the  LLCs  without  automation. 

Lesson  #2  Implications 

Lesson  #2  (Person  in  the  Loop)  indicates  that  LLC  self-assessment  is  far  from  becoming 
an  automated  process.  A  person  is  required  to  (1)  determine  the  minor  modifications  to  the  LLC 
Assessment  Framework  for  particular  applications;  (2)  develop  or  revise  measures  (or  decide  to 
use  the  measures  as  is);  (3)  conduct  data  collection,  analysis,  and  interpretation;  and  (4) 
communicate  results  to  others.  To  perform  these  functions,  this  person  (or  people)  must  have  a 
working  knowledge  of  adult  learning  theory  and  practice,  instructional  design,  educational 
technology,  research  design,  and  elementary  statistics. 

Automating  the  assessment  process,  at  least  in  the  early  stages  of  the  lifelong  learning 
initiative,  not  only  would  not  be  feasible  given  the  rapid  rate  of  change,  the  nascent  state  of 
metrics  and  measures,  and  widely  distributed  sources  of  assessment  data,  but  also  would  lack  the 
diagnostic  detail  necessary  to  understand  the  basis  of  shortfalls  and  stimulate  growth.  For  this 
reason  alone,  rapid  advancement  toward  automation  at  this  time  should  not  be  encouraged. 
Rather,  the  time  and  effort  required  to  design  automated  data  collection  should  be  invested  in 
developing  the  relationships  and  research  methods  necessary  to  integrate  the  diverse  sources  of 
information  about  LLC  effectiveness  and  impact.  Automation  then  could  be  used  to  facilitate  an 
existing  process  rather  than  to  determine  that  process. 
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Lesson  #3  Implications 

The  key  implication  of  Lesson  #3  (Face  Time)  is  that  applying  the  LLC  Assessment 
Framework  must  involve  a  significant  amount  of  time  devoted  to  being  physically  present  at  the 
proponent  schoolhouse  supported  by  the  LLC.  Visits  to  the  schoolhouse  must  involve  face-to- 
face  contact  with  key  stakeholders  and  other  providers  in  which  interpersonal  exchange  fosters 
trust  and  mutual  understanding.  Time  spent  at  the  schoolhouse,  particularly  if  the  assessor  is  only 
a  temporary  visitor,  should  leverage  the  close  proximity  of  stakeholders,  using  the  opportunity  to 
ensure  that  a  large,  representative  sample  of  people  is  contacted. 

Lesson  #4  Implications 

It  should  not  be  concluded  from  Lesson  #4  (Field  Work)  that  the  unit  readiness  metrics 
specified  in  the  LLC  Assessment  Framework  cannot  be  addressed.  Rather,  the  necessary 
authority,  need  to  know,  social  connections,  and  outreach  process  must  be  in  place  to  make 
readiness  data  collection  feasible  and  cost-effective.  The  existing  connections  between  field  units 
and  LLC  staff  who  provide  On-Demand  Training  could  facilitate  data  collection,  and  did,  in  the 
present  research.  In  addition,  LLC  staff  processes  could  be  tailored  to  develop  a  formal  method 
for  maintaining  such  contact.  Closer  coordination  at  the  program/organizational  level  between 
training  and  personnel  managers  could  facilitate  the  data  sharing  necessary  to  determine  LLC 
impact.  Research  conducted  by  institutions  closely  affiliated  with  the  Army  should  be  explicitly 
devoted  to  identifying  the  necessary  means  to  capture  readiness  data  from  the  field  and  the 
challenges  to  applying  these  methods. 

A  Proposed  LLC  Self-Assessment  Plan 

As  alluded  to  above,  a  successful  LLC  self-assessment  program  should  feature  the 
following  characteristics: 

o  A  staff  of  people  with  working  knowledge  of  (ideally  experts  in)  adult  learning  theory 
and  practice,  educational  technology,  instructional  design,  research  design,  and 
elementary  statistics  to  design  and  oversee  the  assessment  process; 

o  An  assessment  process  that  involves  significant  face  time  between  assessors, 
stakeholders,  and  other  providers;  and 

o  Sufficient  authority  and  social  networks  to  orchestrate  extensive  field  data  collection. 

The  program  also  must  be  resource-efficient,  chiefly  by  leveraging  existing  resources, 
capitalizing  on  previous  work,  re-using  assessment  instruments,  and  using  automation,  where 
feasible. 

The  LLC  self-assessment  program  requires  these  features  in  order  to  (1)  maintain 
knowledge  of  the  rapid,  frequent  changes  to  LLC  characteristics;  (2)  determine  whether 
modifications  to  the  LLC  Assessment  Framework  are  necessary  to  conduct  assessment;  (3) 
develop,  as  needed,  new  metrics  and  measures;  (4)  conduct  data  collections;  (5)  analyze, 
interpret,  and  disseminate  findings;  and  (6)  provide  actionable  recommendations  for  future  LLC 
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growth.  Figure  3  below  shows  the  general  layout  of  a  proposed  LLC  self-assessment  plan,  which 
is  discussed  in  more  detail  below. 


Figure  3.  Schematic  of  the  Proposed  LLC  Assessment  Plan 


External  Assessors 

The  people  ultimately  responsible  for  conducting  LLC  self-assessment  should  be 
disinterested  parties,  independent  of  the  lifelong  learning  initiative.  They  should  not  be  employed 
by  an  organization  whose  access  to  funding  or  other  resources  is  linked  to  assessment  findings. 
Assessment  personnel  also  should  not  be  the  same  people  responsible  for  making  decisions  about 
funding  or  the  allocation  of  other  resources  to  the  LLC.  Maximizing  the  independence  of 
assessors  in  this  way  reduces  the  likelihood  that  biases  are  consciously  or  (more  likely) 
unconsciously  used  to  guide  the  investigative  “spotlight”  or  the  interpretation  of  data. 

In  the  proposed  LLC  self-assessment  plan,  external  assessors  are  assessment  specialists 
independent  of  the  proponent  and  LLC  program  management  who  plan  assessment,  collect  data. 
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analyze  findings,  and  present  results.  They  use  the  LLC  Assessment  Framework  as  a  basis  for 
designing  their  assessment  plan,  as  well  as  other  resources,  as  necessary.  For  instance,  the 
Advanced  Distributed  Learning  Co-Laboratory  (ADL  Co-Lab)  provides  training  and  other 
support  materials  for  educational  program  evaluation.  These  materials  include  a  worksheet  for 
planning  training  evaluation,  sample  survey  questions  to  capture  student  and  instructor  reactions 
to  training,  and  slides/handouts  detailing  the  training  evaluation  process.  Although  the  sample 
survey  questions  relate  to  only  one  aspect  of  the  LLC  Assessment  Framework,  the  planning 
worksheet  and  evaluation  process  slides/handout  could  be  used  to  assist  with  assessment  more 
broadly.  All  of  these  documents  may  be  retrieved  from  the  ADL  Co-Lab  by  creating  an  account 
with  adlcommunity.net  and  searching  the  contents  of  the  Research  and  Evaluation  folder. 

Another  useful  resource  for  assessors  is  TRADOC’s  Pamphlet  350-70-4  (2004)  and 
Accreditation  Standards  and  Guide  (2007).  The  pamphlet  details  the  assessment  process, 
including  how  to  plan  assessment,  collect  data,  analyze  findings,  and  disseminate  results.  The 
evaluation  guide  for  accreditation  presents  3 1  standards  for  evaluating  proponent  instruction  as 
well  as  detailed  instructions  for  applying  them.  Several  of  these  standards  overlap  with  the  LLC 
Assessment  Framework  methods.  Hays,  Stout,  and  Ryan-Jones  (2005)  presents  assessment 
checklists  to  be  used  when  assessing  CBTAVBT.  Although  none  of  these  documents  alone  is 
fully  applicable  to  assessing  LLCs,  these  resources,  combined  with  the  LLC  Assessment 
Framework  and  associated  reports,  should  provide  sufficient  guidance  for  knowledgeable 
individuals  to  conduct  a  program  evaluation  consistent  with  the  Army’s  expectations. 

External  assessors  may  come  from  within  the  government  or  outside  of  it.  One  option  for 
external  assessment  provided  by  the  government  is  the  ADL  Co-Lab.  The  ADL  Co-Lab  assists  in 
evaluating  the  efficiency  and  effectiveness  of  other  initiatives  that  fall  under  the  general  lifelong 
learning  rubric,  including  the  U.S.  Joint  Forces  Command  individual  augmentee  training 
program  and  the  Defense  Ammunition  Center’s  internship  training  program,  among  others. 
Another  external  assessor  within  the  government  could  be  the  TRADOC  Analysis  Center. 
Assessors  from  outside  the  government  may  come  from  the  plethora  of  contractors  who 
specialize  in  instructional  design  and  educational  program  evaluation. 

External  assessors  should  be  capable  of  recognizing  the  larger  social  context  in  which 
assessment  takes  place.  People  with  this  ability  are  better  able  to  understand  (1)  the  concerns  that 
stakeholders  and  other  providers  will  have  about  being  assessed;  (2)  the  methods  for  building 
trust  and  exchange;  (3)  the  organizational  context  that  places  external  constraints  on  LLC 
impact;  and  (4)  the  importance  of  framing  results  in  a  way  that  is  true  to  the  findings  but 
satisfactory  to  the  stakeholders  involved.  Sensitivity  to  the  larger  social  context  prepares 
assessors  to  apply  their  assessment-related  expertise  in  a  way  that  will  produce  the  most 
meaningful  findings,  accepted  results,  and  actionable  recommendations.  Understanding  the 
larger  context  would  be  greatly  enhanced  by  an  effective  LLC  Staff  Augmentee,  whose  role  is 
described  below. 

LLC  Staff  Augmentee 

The  proposed  LLC  Staff  Augmentee  plays  the  critical  role  in  LLC  assessment  by 
facilitating  the  connection  between  external  assessors,  LLC  program  management,  proponent 
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stakeholders,  external  data  sources  (e.g.,  personnel  databases),  and  external  stakeholders  (i.e., 
field  units).  Like  the  external  assessor,  this  person  has  expertise  in  the  areas  of  adult  learning 
theory,  technology-assisted  instruction,  and  educational  program  evaluation.  The  LLC  Staff 
Augmentee  is  situated  with  the  proponent,  but  is  a  government  employee  of  the  larger  LLC 
program.  This  person  therefore  would  have  the  expertise  and  access  necessary  to  serve  as  a 
multi-party  liaison  who  is  capable  of  coordinating  closely  with  the  LLC  technical  staff, 
schoolhouse  faculty  and  course  developers,  subject  matter  experts,  proponent  quality  assurance 
personnel,  and  field  unit  trainers.  This  person  would  be  able  to  reach  back  to  the  program  level 
of  LLC  management  to  facilitate  access  to  external  database  information  and  field  unit 
leadership,  which  will  enable  data  collection  that  is  external  to  the  proponent  schoolhouse.  As  an 
employee  at  the  program  level,  the  LLC  Staff  Augmentee  also  would  be  responsible  for  ensuring 
that  the  assessment  and  evolution  of  a  particular  LLC  feeds  the  larger  process,  including  the 
production  of  generalizable  assessment  methods  and  the  development  of  a  shared  knowledge 
base  of  LLC  program  effectiveness. 

The  knowledge  and  close  coordination  of  the  LLC  Staff  Augmentee  would  facilitate  data 
collection  by  assisting  external  assessors  in  identifying  stakeholders  (inside  and  outside  of  the 
schoolhouse),  procuring  archival  data,  and  achieving  access  to  schoolhouse  courses,  field  unit 
trainers,  and  personnel  and  training  bases.  The  LLC  Staff  Augmentee  also  would  provide 
external  assessors  with  the  current  technical  status  of  the  LLC,  as  well  as  the  history  of  changes, 
the  larger  organizational  context,  and  related  documentation.  The  implementation  of  assessors’ 
recommendations  would  be  enabled  by  the  LLC  Staff  Augmentee’s  collaboration  with  proponent 
quality  assurance  personnel  responsible  for  faculty  development.  The  LLC  Staff  Augmentee 
would  contribute  his  or  her  expertise  in  technology-assisted  instruction  to  designing  and 
delivering  faculty  development  at  the  request  of  quality  assurance. 

Quality  Assurance  (Proponent) 

The  expertise  of  proponent  quality  assurance  personnel  should  be  leveraged  to  assist  with 
data  collection  both  internal  and  external  to  the  schoolhouse.  These  individuals  have  significant 
experience  with  conducting  assessment  of  Army  training  initiatives  and  have  ready  access  to 
existing  resources  that  could  support  the  LLC  evaluation  effort.  Ongoing  quality  assurance 
studies  should  be  brought  to  the  attention  of  external  assessors  by  the  LLC  Staff  Augmentee  so 
that  efforts  are  not  duplicated  and  time  is  saved  by  re-using  assessment  processes.  Moreover,  the 
LLC  Staff  Augmentee  should  identify  recurring  LLC  assessment  targets  and  work  with  quality 
assurance  to  include  these  as  part  of  their  investigation  of  proponent  effectiveness.  Additional 
contracted  personnel  may  be  required  to  assume  added  quality  assurance  duties,  but  quality 
assurance  leaders  have  experience  recruiting,  selecting,  and  training  educational  assessors  and 
may  be  considered  more  independent  than  members  of  an  LLC  staff  who  might  be  considered  as 
alternatives. 

LLC  Program  Management  (TRADOC) 

The  role  of  LLC  program  management  in  the  proposed  assessment  plan  is  to  ensure  that 
lifelong  learning  assessment  and  development  are  tightly  coupled  through  coordinated  processes 
across  the  LLCs.  These  processes  should  not  necessarily  be  standardized,  given  the  unique  needs 
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of  each  LLC,  but  they  should  avoid  duplication  of  effort,  collectively  paint  the  “big  picture”  of 
the  success  of  lifelong  learning,  and  ultimately  facilitate  the  connection  between  the  lifelong 
learning  initiative,  the  proponent  schoolhouses,  and  the  field  Army.  The  LLC  program 
management  would  accomplish  such  synchronized  diversity  by  maintaining  reporting  authority 
over  the  LLC  Staff  Augmentees  but  situating  them  with  the  proponents.  Augmentees  would  be 
able  to  develop  the  critical  relationships  necessary  to  enable  lifelong  learning  but  also  would 
have  the  obligation  to  coordinate  with  the  augmentees  of  other  LLC  staffs  and  report  to  the  LLC 
program  management  about  the  success  of  the  larger  lifelong  learning  effort.  In  addition,  the 
organization-level  connections  of  the  LLC  program  management  with  the  Human  Resources 
Command  and  the  key  leadership  of  field  units  would  facilitate  the  augmentee  reach  back  for  the 
collection  of  data  external  to  the  proponent  schoolhouse. 

Reconsidering  the  Purpose  of  the  LLC  Staff 

A  critical  characteristic  of  the  proposed  LLC  self-assessment  plan  is  that  it  involves  a 
slight,  but  important  modification  to  the  envisioned  purpose  of  the  LLC  staff.  The  proposed  plan 
is  predicated  on  the  assumption  that  the  success  of  lifelong  learning  is  as  much  a  function  of 
social  networks  as  it  is  of  technical  outputs  and  that  the  LLC  staff  could  be  uniquely  situated  to 
provide  both.  Currently,  the  role  of  the  LLC  staff  is  considered  to  be  largely  technical,  involving 
the  construction  and  maintenance  of  the  LLC  portals  and  the  provision  of  technical  support.  In 
contrast,  responsibility  is  somewhat  more  diffuse  and  less  well  articulated  for  ensuring  (1)  that 
relationships  are  developed  with  proponent  stakeholders  to  rapidly  produce  effective  training  on 
demand  and  to  conduct  meaningful  assessment;  (2)  that  proponent  faculty  and  course  developers 
understand  how  to  leverage  technology  for  assisting  adult  learners;  (3)  that  LLC  functionality  fits 
into  the  larger  context  of  not  only  the  proponent’s  objectives,  but  those  of  organizational  Army; 
and  (4)  that  a  body  of  knowledge  is  developed  to  track  and  facilitate  the  growth  of  the  lifelong 
learning  concept.  Carrying  out  this  responsibility  requires  the  cooperation  and  coordination  of  a 
diverse  set  of  organizations  that  may  or  may  not  be  aware  (or  supportive)  of  their  role  in 
enabling  lifelong  learning.  A  central  source  of  strong  leadership  that  understands  the  needs  of  the 
proponent,  the  LLC  program  management,  and  the  larger  context  is  needed  to  maximize  impact. 

LLC  technical  staffs  as  currently  constructed  do  not  have  the  personnel,  expertise,  or 
authority  to  take  on  these  challenging  and  complex  roles.  Yet,  as  the  findings  of  the  LLC 
assessment  research  program  suggest  to  date,  these  roles  are  essential  to  ensuring  the  success  of 
lifelong  learning  by  strengthening  the  link  between  technical  outputs,  external  factors,  and 
impact.  The  addition  of  the  LLC  Staff  Augmentee  in  the  proposed  assessment  plan  is  not  simply 
a  recommended  method  for  conducting  LCC  assessment,  but  also  represents  an  important  first 
step  in  acknowledging  and  enacting  a  more  proactive  role  of  LLC  program  management  in 
equipping  LLC  staffs  to  achieve  the  larger  goals  of  lifelong  learning. 

CONCLUSIONS 

The  present  research  explored  the  generalizability  of  a  framework  for  assessing  the 
effectiveness  and  impact  of  LLCs,  the  concrete  instantiation  of  the  U.S.  Army’s  lifelong  learning 
concept.  The  analysis  presented  in  this  report  involved  a  substantial  increase  in  the  types  of 
instruction  and  learning  audiences  supported  by  lifelong  learning  than  has  been  conducted 
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previously,  producing  a  more  representative  sample  on  which  to  base  the  framework’s  design.  It 
was  determined  that  some  modification  was  necessary  to  the  assessment  framework  to  address 
the  variety  of  possible  functions  an  LLC  could  provide,  but  that  the  logic  modeling  approach 
remained  a  valid  method  for  conducting  qualitative  causal  analysis. 

A  secondary  purpose  of  the  present  research  was  to  conduct  an  assessment  of  the  Fort 
Gordon  LLC.  The  results  of  this  assessment  indicated  that  although  the  expected  outputs  for  this 
LLC  generally  were  delivered,  external  factors  moderated  the  link  between  outputs  and 
outcomes  at  all  levels  (individual,  unit,  and  organizational).  That  is,  in  the  absence  of  controlled 
experimental  study,  inferences  could  be  made  regarding  the  determinants  of  program  strengths 
and  weaknesses  by  using  a  qualitative  causal  model.  Shaping  the  influence  of  external  factors 
required  closer  coordination  among  the  LLC  staff  and  proponent  offices  as  well  as  closer 
coordination  among  decision  makers  at  the  organizational  and  Army  level. 

A  third  purpose  of  this  research  effort  was  to  specify  an  assessment  plan  that  would 
enable  LLCs  to  adopt  the  revised  LLC  Assessment  Framework  to  conduct  self-assessment.  The 
design  of  this  plan  was  based  on  lessons  learned  in  the  present  investigation  as  well  as  previous 
ARI  LLC  assessment  research  (Cianciolo,  2007).  A  key  feature  of  this  plan  is  the  emphasis 
placed  on  reconsidering  the  role  of  LLC  staffs  such  that  they  serve  a  critical  social  as  well  as 
technical  function.  The  self-assessment  plan  addressed  the  need  to  more  closely  coordinate 
efforts  by  situating  program-level  assessment  personnel  at  the  proponent.  Staffing  in  this  way 
would  facilitate  the  social  connections  necessary  to  conduct  assessment  internally  and  externally 
and  would  ensure  that  assessment  findings  were  actively  used  to  enhance  the  larger  lifelong 
learning  initiative. 


Limitations  of  the  Present  Research 

The  limitations  of  the  present  research  involve  the  lack  of  empirical  basis  of  the  Fort 
Gordon  LLC  assessment.  First,  emphasis  on  exploring  methods  to  substitute  for  surveys, 
combined  with  the  short  duration  of  this  research  effort,  reduced  the  representativeness  of  the 
sample  used  in  some  cases  (e.g.,  only  three  SCCC  classes  were  observed,  only  three  25B10 
instructors  were  interviewed).  In  these  instances,  converging  evidence  was  used  to  support  lack 
of  data,  but  further  study  is  warranted.  Where  representative  sampling  was  possible  (e.g.,  as  in 
discussion  forum  analysis),  it  was  conducted.  Moreover,  in  several  instances,  the  relatively  small 
sample  actually  was  the  population  (e.g.,  25B10  was  the  only  MOSQ  course  delivered  both  in 
the  schoolhouse  and  to  non-resident  students). 

Second,  where  comparisons  across  conditions  were  of  interest  (e.g.,  training  effectiveness 
with  and  without  simulations),  lack  of  authority  to  manipulate  treatment  conditions  prevented 
controlled  experimental  study.  Where  possible,  archival  data  and/or  quasi-experimental  designs 
(e.g.,  as  in  the  SCCC)  were  used-and  the  qualitative  causal  model  provided  a  great  deal  of 
converging  evidence— but  systematic  study  would  provide  a  stronger  basis  for  conclusions. 

An  additional,  important  limitation  of  the  present  research  was  the  lack  of  external  data 
that  could  provide  information  about  the  impact  of  lifelong  learning  on  unit  readiness. 
AUTOGEN  surveys,  as  well  as  surveys  created  as  part  of  this  research  effort,  suffered  from 
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insufficient  response  rates.  Phone  interviews  with  unit  trainers  who  established  Unit  Universities 
were  helpful,  but  failed  to  provide  detailed  accounts  of  the  impact  of  lifelong  learning.  The 
enthusiasm  of  unit  trainers  for  LLCs  was  great,  but  direct  contact  with  and  observation  of  these 
trainers’  units  is  necessary  to  understand  the  concrete  reality  of  how  training  on  demand  has 
changed  the  readiness  posture  of  units  in  the  field. 

Recommendations  for  Future  Research  and  Practice 

Future  research  should  focus  on  identifying  and  applying  the  means  necessary  to  conduct 
external  data  collection.  Visits  to  line  units  actively  using  lifelong  learning  should  be  paid  in 
order  to  observe  directly  the  impact  that  anytime,  anywhere  instruction  has  on  performance.  Face 
time  with  these  units  also  would  enable  the  collection  of  comprehensive  interviews  and  focus 
groups  to  gather  detailed  information  for  developing  additional  outcome  metrics.  Such 
investigation  would  reveal,  for  example,  the  acceptable  amount  of  lag  between  the  request  for 
and  delivery  of  On-Demand  Training.  Physical  presence  with  the  units  also  would  reveal 
additional  external  factors  that  influence  the  adoption  of  lifelong  learning  and  potential  methods 
for  addressing  these  factors  and  expanding  LLC  marketing  outreach.  The  schoolhouse 
environment  and,  to  some  extent,  the  distance  learning  setting  is  fairly  well  understood,  but  the 
operational  learning  environment  remains  largely  unexplored. 

Future  research  also  should  explore  and  identify  the  additional  modes  of  instruction  and 
learning  audiences  that  are  being  or  will  be  addressed  by  current  and  envisioned  LLCs.  Although 
a  wide  sampling  of  educational  opportunities  was  featured  in  this  report  and  in  Cianciolo  (2007), 
there  is  sufficient  reason  to  believe  that  this  sample  was  not  comprehensive.  This  research  could 
be  conducted  by  reviewing  each  LLC’s  planning  documentation  and  conducting  site  visits  to 
each  proponent  to  interview  key  stakeholders. 

Future  practice  must  ensure  that  the  critical,  proactive  social  role  of  the  LLC  staff  be 
acknowledged  and  actively  developed.  Initiating  self-assessment  as  envisioned  in  this  report 
would  represent  a  significant  step  in  this  direction.  Future  research  supporting  this  effort  should 
explore  the  particular  social  networks  and  processes  necessary  to  enable  effective  LLC 
functioning  at  the  regional  and  enterprise  level,  as  well  as  at  the  proponent  level.  Changes  to 
technical  support  structures  must  retain  the  critical  coordinative  and  collaborative  roles  served  by 
a  fully  functional  LLC  staff. 


Closing  Thoughts 

In  summary,  the  present  investigation  attempted  to  achieve  three  goals:  (1)  to  examine 
the  generalizability  of  the  LLC  Assessment  Framework  put  forth  in  previous  ARI  research;  (2)  to 
apply  the  revised  framework  to  assessing  the  Fort  Gordon  LLC;  and  (3)  to  explore  the 
requirements  for  enabling  LLC’s  to  conduct  their  own  assessment  in  the  future.  The  findings  of 
this  research  were  that  the  assessment  framework  was  generalizable,  but  required  modification  in 
order  to  address  the  broad  range  of  training  and  education  strategies  supported  by  the  Fort 
Gordon  LLC.  It  is  expected  that  this  variety  of  strategies  represents  the  majority  of  learning 
approaches  that  will  be  taken  by  other  current  and  future  LLCs,  but  some  modification  to  the 
framework  will  be  necessary  for  each  assessment  that  is  conducted.  The  Fort  Gordon  LLC  was 
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found  generally  to  be  accomplishing  its  output  goals  but  that  external  factors  sometimes 
moderated  the  link  between  outputs  and  expected  outcomes.  Recommendations  for  enhancing 
outcomes  and  for  conducting  future  LLC  assessment  emphasized  the  importance  of  coordination 
among  the  LLC  technical  staff,  proponent  stakeholders,  the  LLC  program  management,  and 
other  stakeholders  at  the  Army  level.  LLCs  represent  the  future  of  adaptive  training  and 
education,  and  it  is  hoped  that  these  findings  will  foster  continued  growth  and  the  broad  impact 
envisioned  for  the  lifelong  learning  initiative. 
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APPENDIX  A  -  ACRONYMS 


ADL  Co-Lab 
ANCOC 

AOT 

ARI 

ATRRS 

AUTOGEN 

Advanced  Distributed  Learning  Co-Laboratory 

Advanced  Non-Commissioned  Officer  Course 
assignment-oriented  training 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Army  Training  Requirements  and  Resources  System 
automatic  survey  generation 

BCKS 

BNCOC 

BOLC 

Battle  Command  Knowledge  System 

Basic  Non-Commissioned  Officer  Course 

Basic  Officer  Leadership  Course 

CBT 

computer-based  training 

DL 

distance  learning 

HTRC 

high-tech  regional  center 

JNN 

joint  network  node 

LLC 

LTC 

lifelong  learning  center 
lieutenant  colonel 

MANSCEN 

MOS 

MOSQ 

MTT 

Maneuver  Support  Center 
military  operating  specialty 
military  operating  specialty  qualified 
mobile  training  team 

NCO 

non-commissioned  officer 

SCCC 

SOP 

Signal  Captains  Career  Course 
standard  operating  procedure 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

UIT 

University  of  Information  Technology 

WBT 

web-based  training 
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APPENDIX  B  -  REVISED  LLC  ASSESSMENT  FRAMEWORK 
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Data  Collection 

User  survey  +  coordination 
with  technical  staff  (to  locate 
operational  users  who  have 
requested  technical  support) 

System  analysis  (help  desk 
data  and  website  analysis)  + 
Technical  staff  survey 

System  analysis  -  help  desk 
data 

System  analysis  -  help  desk 
data 

. 

User  survey  (multiple  items) 
or  Focus  groups 

User  survey  (multiple  items) 

or  Focus  groups 

System  analysis  -  help  desk 

data  and  website  analysis 

Applicable  To 

Discussion  forums,  On- 
Demand  Training,  Simulation 
downloads 

Discussion  forums,  On- 
Demand  Training,  Simulation 
downloads 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Metric 

Perceived  utility  of 
operational  technical 
support 

%  repeated  operational 
questions  that  are 
integrated  into  help 
desk  FAQ  updates  or 
other  forums 

%  technical  trouble 
tickets  (phone  and 
walk-in)  closed  within 

Va  hour  of  posting 

%  technical  trouble 
tickets  (email)  closed 
within  Vi  hour  of 
posting 

Perceived  accessibility/ 
responsiveness  of 
technical  support 

Perceived  utility  of 
technical  support 

%  repeated  technical 
trouble  tickets  (i.e., 
multiple  users  with 
same  request)  that  are 
integrated  into  help 
desk  FAQ  updates 

Metric  Category 

Provide  support  for 
answering  Field  Armv 

users’  operational 
questions 

Provide  technical  support 
to  students,  instructors, 
and  course  developers 

LWN  1 

X 

X 

X 

MANSCEN  1 

X 

X 

X 

LVNW  1 

X 

X 

X 

X 

X 

Data  Collection 

System  analysis  and  Technical 
staff  interviews 

System  analysis  and  Technical 
staff  interviews 

User  survey  (multiple 
indicators  to  include 
management  of  user  accounts, 
creation  of  initial  sites  and 
templates,  archive 
management,  etc.) 

System  analysis  +  Technical 

staff  survey  or  Focus  groups 

Instructor  survey  (1  or  2  items 

capturing  whether  content  is 

available  in  Bb  when  needed) 

or  Focus  groups 

Applicable  To 

< 

5 

3 

Leader  education 

Leader  education 

Metric 

%  repeated  technical 
trouble  tickets  (i.e., 
multiple  users  with 
same  request)  that  are 
integrated  into 
application 

customization  updates 

Vo  tecnmcal  problem 
tickets  that  resulted 
from  ineffective 
technology 
management  (e.g., 
account  problems, 
template  bugs,  etc.) 

Perceived 
responsiveness  and 
competency  of 
technical  staff  to 
customization  requests 

%  content  migrated 
across  applications 
within  24  hours 

Perceived  speed  of  the 
content  migration 
process 

Metric  Category 

Perform  technology 
setup,  integration, 
customization,  and 
management  of  LLC 
components 

Migrate  course  content 
across  LLC  components 
(e.g.,  from  SharePoint  to 
Bb) 
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X 

X 

Data  Collection 

System  analysis  +  Instructor 
survey  or  Focus  groups 

System  analysis  +  Instructor 
survey  or  Focus  groups 

System  analysis  +  Instructor 
survey  or  Focus  groups 

Instructor  survey  +  Tech  staff 
survey  (single  item)  or  Focus 
groups 

Instructor  survey  (single  item) 
or  Focus  groups 

Instructor  survey  (single  item) 

or  Focus  groups 

Applicable  To 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Leader  education,  MOSQ 
instruction  (resident,  non¬ 
resident,  and  simulation- 
supported,  and  AOT  format) 

Metric 

%  instructors  who  use 

Bb  to  administer  course 

exams 

%  instructors  who 
leverage  Bb  exam 
functions  to  provide 
immediate  feedback 
and  remedial  instruction 

%  instructors  using  Bb 
to  report  student  grades 
to  administrators 

Presence  of  formal 

SOPs  for  leveraging  the 
capabilities  of  the 
software  applications 
for  course 
administration  and 
facilitation  (Y/N) 

Presence  of  informal 
SOPs  for  leveraging  the 
capabilities  of  the 
software  applications 
for  course 
administration  and 
facilitation  (Y/N) 

%  of  early  adopters 
involved  in  SOP 
development 

Metric  Category 

Evaluate  student  progress 
and  report  grades  to 
school  administrators 

(Early  adopters)  Lead  the 
development  of  SOPs  for 
leveraging  the 
capabilities  of  system 
components  (e.g., 
collaborative  capabilities 
in  Bb) 
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MANSCEN 
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X 

LVNW  1 

Data  Collection 

Archival  data  +  Coordination 
with  courseware  production 
team  +  Coordination  with 
outside  contractors 

Contractor  survey  (multiple 
items  to  include  matters  of 
timeliness,  clarity  of 
communication,  alignment  of 
expectations,  etc.)  or  Focus 
groups  +  Leadership  survey 
(items  pertaining  to 

complaints,  disputes,  lags  in 

contracting) 

Applicable  To 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Metric 

Ratio  of  time  spent 
between  courseware 
submission  to  ATSC 
and  acceptance 
(SCORM  conformance 
and  TRADOC  350-70 
Regulation 

conformance)  for  LLC- 
overseen  CBTAVBT 

versus  non-overseen 
CBTAVBT  (value  <  1 
indicates  time  savings 
in  the  submission- 
acceptance  process  for 
LLC-overseen  CBT/ 
WBT) 

Perceived  quality  of 
contract  oversight 

Metric  Category 

Provide  contractual, 
technical,  and 
educational/quality 
oversight  of  contractor- 
developed  CBTAVBT 

LWN 

MANSCEN 

X 

X 

X 

X 

LVNW 

Data  Collection 

System  analysis  (using  a 
checklist  for  type  of  materials 
present,  if  database  exists) 

System  analysis 

Courseware  production  team 

survey  or  Focus  groups  + 

System  analysis 

Courseware  production  team 

survey  or  Focus  groups  + 

System  analysis 

Applicable  To 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Metric 

Presence  (Y/N)  of 
database  containing 
documentation  on 
CBT/WBT 
technologies, 
capabilities,  and 
techniques,  CBT/WBT 
lessons  learned, 
scholarly  literature  on 
adult  education  and 
educational  technology, 
and  lifelong  learning 
strategies 

%  ot  materials  m  the 
database  that  are 
current  (written  within 
the  past  2  years; 
expectation  is  for  a 
roughly  horizontal 
distribution) 

%  courseware 
production  team 
members  who  access 
database  materials 
(weekly,  monthly,  prior 
to  new  project) 

%  ot  courseware 
production  team 
members  who  regularly 
(daily,  weekly, 
monthly,  etc.) 
contribute  database 
materials 

Metric  Category 

Maintain  a  database  of 
CBT/WBT  technologies, 
capabilities,  and 
techniques 
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Data  Collection 

System  analysis 

Courseware  production  team 
survey 
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%  5 

System  analysis  +  Leader 
interview 

Instructor  and  developer 
survey  (single  item)  or  Focus 
groups 

Archival  data  analysis  (needs 
assessment  review) 

Leader  interview  +  instructor 

and  developer  survey  (single 

item)  or  Focus  groups 

Leader  interview 

Applicable  To 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

% 

3  • 

% 

3 

3 

Metric 

Frequency  of  updates  to 
the  database  (daily, 
weekly,  monthly,  etc.) 

Perceived 

accessibility/utility  of 
the  database 

Activities:  Leadership 

Metrics  answer  the  Question:  Is  the  lenderchin  <••••» 
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Formal  user  and 
stakeholder  needs 
assessment  conducted 
(Y/N) 

Stakeholder  perceptions 
of  inclusion  in  the 
decision-making 
process 

rresence  ti/iN)  m 
needs  assessment  of 
external  factors 
recognized  as  potential 
moderators  of  success 

Frequency  of 
leader/stakeholder 
interactions  during  LLC 
planning  and 
implementation 

Perceived  involvement 
in  the  design  of  his/her 
proponent’s  LLC  (as 
opposed  to  design 
coming  from  top-down 
or  strictly  bottom-up 

Metric  Category 

Maintain  a  database  of 
CBT/WBT  technologies, 

capaouuies,  ana 
techniques 

Conduct  user  and 

ntnl _ l_  1  J  _  J 

Mduciiuiucr  neeas 

assessment 

Provide  vision 

m 

© 


X 


X 


X 


X 


CL  CL 


V 

5 

+-> 

.s 

fc 

TD 

CO 


E 

E 

<d 

s 

u 

C 

■§ 


3 

o 

o 

Uh 

u 

o 

>> 

<D 


c/3 

Cm 

l 


a  i 

C/5  o 

O  c 


c 
-  u 

o  s 

~  C/3 

3  8 

O  w 

1.1 


I  ,2 


S  s  JS  t 
sO 

e  < 


v  5 

3  E 
2  § 

J3 .1 


c/3  T3 

■g  « 

«0  -q 

«J*  « 

c  € 
U  o 
•g  a 
•g  p. 

CD  3 

U,  V) 


00 
u  S 

*  s 


T3  — 

CO  C/3 

w  «  c 

*  « Jl 

Cm  O. 
°  "0 
<L>  <D 

o  a 
a  o 

X  8 

<L>  > 


C 

6 

a 

o 

a 

o 


S  S. 


«  v 

cu  -a 


&  v 

8 1 
“>2  o 

2  £  J 
■S  o  J 


3 
8 

5  4e  %  c 
—  v  «  a> 

i'S  5  *2 

^  93  tfi  > 

13  2^-2  J  g 
"ug-ga 

«  22  ~  «  g 

3  s  §>s!  j 


o 

a 


a 

«  -®  I  u 
of  IS5 

-  w  ^  (X  c 

£  $  2  53  3 
JD  «  ^3  —  5 

2  s  wi2  ~  a 
S  a  -c  *  -o 
a  S.au  s 
ogf-ja 


0II 
is  2 

i>3  o 

75  CD 

-D  j^<+2 
0>  *— 

5  2  | 
•o  y  g 
.  .  *- > 
O  X 
CD  <D 


a  SI 


»  2 

•s  ^ 

<l>  ° 

ti  c/3 
&  C/3 


8  C 


M  S 

u 

>,  g 

a  «S 

S  V 


•Q  b 
&  <2 
*3  «« 

to  TJ 
oo  2 

e  -5  ^ 
2  g  | 

o  c  2 

to  _,  X 
U  T3  ^ 

u  c  00 

c  3.9 

§11 

-J  00  JC 


a 

o 


<u 

T3 

’> 

O 


»«2 
y  a  ,2 
£  Sc 

3  -g  T3 

gSg 

CD  «-»  CO 
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X 

X 
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X 

Data  Collection 

Course  developer  survey  or 
Focus  groups 

Technical  staff  survey  or 

Focus  groups 

Courseware  developer  survey 
or  Focus  groups 

User  survey  or  Focus  groups 

Special  purpose  measure  (% 
agreement  among  survey 
responses) 

Special  purpose  measure 

(checklist  of  marketing 

activities,  to  include  mass 

emails,  briefings, 

advertisements,  etc.) 

Leader  interview  +  Personnel 

interview  or  Focus  groups 

Leader  interview  +  Personnel 

interview  or  Focus  groups 

Applicable  To 

Leader  education 

% 

Leader  education,  MOSQ 
instruction  (resident  and  non¬ 
resident) 

On-Demand  Training, 
Discussion  forums 

% 

3 

3 

% 

Metric 

%  course  developers 
who  understand  the 
“commander’s  intent” 
for  the  LLC 

%  technical  staff  who 
understand  the 
“commander’s  intent” 
for  the  LLC 

%  courseware 
developers  who 
understand  the 
“commander’s  intent” 
for  the  LLC 

%  field  Army  users 
who  understand  the 
"commander's  intent" 
for  the  LLC 

Presence  of  shared 
understanding  among 
stakeholders  of  the 
purpose  and  status  of 
the  LLC 

%  marketing  LLC 
activities  conducted 

Adequacy  of  funding 
for  LLC 

Adequacy  of  personnel 
(#s)  for  LLC 

Metric  Category 

Communicate 
vision/market  LLC 
concept  to  stakeholders 
(including  course 
developers,  instructors, 
courseware  developers, 
technical  staff,  students, 
and  field  Army) 

Procure  resources  to 
maintain/update  the  LLC 

LWN 

X 

X 

X 

X 

X 

MANSCEN 

X 

X 

X 

X 

X 

LVNW 

X 

X 

X 

X 

X 

Data  Collection 

System  analysis  -  Coordination 
with  technical  staff 

User  survey  or  Focus  groups 

System  analysis  -  Coordination 
with  technical  staff 

System  analysis  -  Coordination 
with  technical  staff 

User  survey  (1  question  asking 

users  to  rate  system 

accessibility)  or  Focus  groups 

Applicable  To 

All 

< 

3 

< 

3 

Metric 

%  operational  time 
spent  in  technical 
outages  or  otherwise 
down  (altogether  and 
broken  down  by 
component 
application) 

user  perceptions  ot 
accessibility 
(altogether  and 
broken  down  by 
component 
application) 

rattem  or  access  as  a 
function  of  time  of 
day  (horizontal  vs. 
bi-modal  vs.  some 
other  distribution) 

frequency  ot  access 
to  system  during 
normal  business 
hours  vs.  off-hours 

/o  users  reporting 
more  than  minor 
difficulty  accessing 
the  system  (broken 
down  by  LLC 
component  and 
location  type) 

Metric  Category 

24/7  Access  to 

System  -  System 
Availability 

24/7  Access  -  Actual 
System  Usage 

LWN 

X 

X 

X 

MANSCEN 

X 

X 

X 

LVNW 

X 

X 

X 

Data  Collection 

System  analysis  (coordination 
with  technical  staff)  +  Archival 
data  (e.g.,  SIDPERS,  to  get 
population  data) 

Archival  data  (programs  of 
instruction  and  course 
crosswalks)  +  System  analysis  - 
Coordination  with  technical 
staff  ("common"  materials  need 
not  match  exactly,  but  must 
enable  students  to  meet  the 
same  learning  objectives) 

System  analysis  +  Archival  data 
(programs  of  instruction  and 
course  crosswalks)  +  Focus 
groups  and/or  Interviews 

Applicable  To 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident),  Simulation 
downloads 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident),  Simulation 
downloads 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident),  Simulation 
downloads 

Metric 

%  users  from  each 
applicable  location 
(schoolhouse, 
satellite  locations, 
TASS  region  sites, 
and  home/deployed; 
the  ideal  balance  at 
any  one  time  is 
representative  of  the 
population) 

%  curriculum 
materials  found  in 

Bb  that  are  common 
across  user  locations 
(schoolhouse, 
satellite  locations, 
TASS  region  sites, 
and  home/deployed) 

Equivalence  of 
learning  experience 
across  resident  and 
non-resident  learners 
(includes  content  and 
instructional  strategy 
equivalence) 

Metric  Category 

Uniform  Access  to 
System  and  Training 
Content 

I.WN  1 

X 

X 

X 

X 

MANSCEN  1 

X 

X 

X 

X 

LVNW  1 

X 

X 

X 

X 

Data  Collection 

Archival  data  (programs  of 
instruction  and  course 
crosswalks)  +  System  analysis  - 
Coordination  with  technical 
staff  (requires  access  to  all 
sections,  teams,  etc.  of  a 
particular  course  at  all 
applicable  locations) 

Archival  data  analysis 
(SIDPERS)  +  System  data 
analysis 

User  survey  (multiple  questions 
corresponding  to  the  different 
uses  of  the  LLC,  e.g.,  course 
instruction  vs.  knowledge 
acquisition  vs.  training)  or 

Focus  groups 

User  survey  (2  questions,  1  each 

for  usability  and  utility,  per 

component)  or  Focus  groups 

Applicable  To 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident),  Simulation 
downloads 

3 

3 

<i 

Metric 

i - - - — — 

#  of  other  course 
materials  (e.g., 
instructor  updates) 
available  to  resident 
students  that  are  not 
present  in  the  LLC 
(or  domain-general 
components  of  the 
LLC)  and  therefore 
not  available  to 
people  at  other 
locations 

%  intended  target 
audience  registered 
to  use  the  system 

%  of  registered  users 
actively  using  the 
system  (as  a  whole 
and  broken  down  by 
LLC  component; 
“actively”  defined  as 
using  the  system  as 
or  more  frequently 
than  alternatives) 

User  perceptions  of 
usability  and  utility 
(altogether  and 
broken  down  by  LLC 
component) 

Metric  Category 

Uniform  Access  to 
System  and  Training 
Content 

Actual  System  Usage 
(General) 

LWN  1 

X 

X 

X 

X 

MANSCEN 

X 

X 

X 

X 

LVNW 

X 

X 

X 

X 

Data  Collection 

User  survey  (1  question  -  also  a 
measure  of  active  use)  or  Focus 
groups 

User  survey  (1+  questions  -  one 
for  each  relevant  activity)  or 
Focus  groups 

User  survey  (1+  questions  -  also 
measures  of  active  use)  or 

Focus  groups 

User  survey  (1  question  -  also  a 

measure  of  active  use)  or  Focus 

groups  +  Coordination  with 

technical  staff  (to  identify  non¬ 

students  or  former  students) 

Applicable  To 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident,  non¬ 
resident,  simulation-assisted, 
and  AOT  format) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident,  non¬ 
resident,  simulation-assisted, 
and  AOT  format) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident,  non¬ 
resident,  simulation-assisted, 
and  AOT  format) 

On-Demand  Training, 
Discussion  forums,  Simulation 
downloads 

Metric 

%  students  who  seek 
alternative  means  to 
get  curriculum 
materials  other  than 
through  the  LLC 
(e.g.,  email 
attachments  or 
thumb  drives  due  to 
limitations  in  access) 

%  students  who 
access  other 
components  of  the 
LLC  (besides  Bb  or 
other  intended 
portal)  to  conduct 
their  studies  or 
collaborate  with 
peers  or  experts 
(captures  breadth  of 
use  of  the  system) 

%  students  who 
primarily  access  non- 
LLC  resources  to 
conduct  their  studies 
or  collaborate  with 
peers  or  experts 

%  registered  non¬ 
students  or  former 
students  who 
primarily  seek 
alternative  means  to 
get  knowledge  or 
training  other  than 
through  reaching 
back  to  the  LLC 

Metric  Category 

Actual  System  Usage 
(Course  Instruction 
Only) 

Actual  System  Usage 
(Alternative  Learning 
Only) 
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X 

Data  Collection 

User  survey  (single  item)  or 

Focus  groups 

Special  purpose  measure 
(fluency  assessment  or 
situational  judgment 
assessment) 

Special  purpose  measure 
(fluency  assessment  or 
situational  judgment 
assessment) 

Special  purpose  measure 

(fluency  assessment  or 

situational  judgment 

assessment) 

Applicable  To 

3 

% 

% 

3 

Metric 

%  users  who  indicate 
that  they  intend  to 
participate/have 
participated  in  work- 
related  education/ 
professional 
development  as  a 
volunteer  (e.g.,  via 
forums)  (relative  to 
pre-  or  non-LLC 
conditions) 

%  users  who 
demonstrate  broad 
thinking  regarding 
sources  of 
information  for 
learning  (relative  to 
pre-  or  non-LLC 
conditions) 

%  users  who 
demonstrate  broad 
thinking  regarding 
the  location  of 
learning 
environments 
(relative  to  pre-  or 
non-LLC  conditions) 

%  users  who 
demonstrate  broad 
thinking  regarding 
sources  of  learning 
authority  (relative  to 
pre-  or  non-LLC 
conditions) 
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Data  Collection 

User  survey  (multiple  items 
each  relating  to  an  aspect  of 
content  delivered  via  the  LLC) 
Interviews  or  Focus  groups 

Instructor  survey  (multiple 
questions,  one  for  each 
administrative  item)  or  Focus 
groups 

Instructor  survey  (single  item) 
or  Focus  groups 

Instructor  survey  (single  item) 
or  Focus  groups 

Instructor  survey  (single  item) 

or  Focus  groups 

Applicable  To 

% 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Metric 

%  users  who  report 
that  they  have  found 
relevant,  useful 
courses,  courseware, 
or  training  materials 
via  the  LLC 

Estimated  % 
reduction  in  time 
spent  in  classroom 
doing  administrative 
tasks,  including 
announcements, 
handouts,  and  testing 
(estimated,  and 
where  applicable) 
using  LLC 

Estimated  % 
increase  in  time 
available  to  assist 
students  having 
difficulties 

%  instructors  who 
report  using 
increased  class  time 
to  enhance 
instruction 

%  instructors  who 
report  using 
increased  class  time 
to  remediate 
problem  students 

Metric  Category 

Enhanced 

Relevance  of 
Training  and 
Educational 

Content 

Enhanced 

Instructional 

Efficiency 
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Data  Collection 

Student  survey  (multiple  items, 
one  for  each  type  of  helping 
behavior)  or  Focus  groups 

Student  survey  (multiple  items 
related  to  different  types  of 
discussion)  or  Focus  groups  + 
System  analysis 

Student  survey  (single  item)  or 
Focus  groups  +  System  analysis 

Student  survey  (to  include  items 

on  conversation  dominance, 

openness  of  instructor,  feelings 

of  trust  in  constructive 

criticism)  or  Focus  groups 

Classroom  observation 

(summarization  of  1009  forms, 

where  applicable) 

Applicable  To 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Leader  education  (resident  and 
non-resident),  MOSQ 
instruction  (resident  and  non¬ 
resident) 

Metric 

%  students  reporting 
a  sense  of 
commitment  to  and 
among  classmates  to 
share  information, 
provide  task  support, 
and  to  provide 
social/emotional 
support  (reference 
criterion  depends  on 
class  size) 

%  students  engaging 
each  other  in 
discussion  of  course- 
related  or  other 
topics  outside  of  the 
classroom  -  w/  and 
w/o  instructor 

Presence  (Y/N)  of  a 
student  leader  who 
collaboratively 
organizes  student 
efforts  (due  date 
lists,  class  calendar, 
etc.) 

%  students  reporting 
freedom  to 
participate  in 
classroom 
(traditional  or 
virtual)  activity 

%  students  engaging 
in  discussion  of 
course  topics  inside 
the  classroom 
(traditional  or 
virtual)  -  w/  and  w/o 
instructor 

Metric  Category 

Presence  of 

learning 

community 
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System  analysis  -  observer 
ratings  of  courseware  based  on 
best  practice  in  instructional 
design  (to  include  levels  of 
interactivity,  assessment, 
feedback,  scaffolding, 
collaborative  learning,  support 
of  social  interaction,  etc.) 

System  analysis  +  Coordination 
with  course  managers 

Special  purpose  measure 
(summary  measure  of  observer 
ratings  of  courseware  based  on 
best  practice,  see  above  metric) 

Applicable  To 

Leader  education  (non¬ 
resident),  MOSQ  instruction 
(non-resident) 

Leader  education  (non¬ 
resident),  MOSQ  instruction 
(non-resident) 

Leader  education  (non¬ 
resident),  MOSQ  instruction 
(non-resident) 

Metric 

%  of  courseware 
modules  comprised 
of  instructional 
strategies/levels  of 
interactivity  that  are 
appropriate  to  lesson 
content 

%  of  educational/ 
training 

requirements  that 
involve 

collective/team  skills 
that  are  taught  using 
a  synchronous 
collaborative 
learning 
environment 
(including  virtual 
classmates/ 
teammates) 

Quality  ratings  for 
CBT/WBT  produced 
or  overseen  by  LLC 
courseware 
production  team 
(relative  to  criterion) 

Metric  Category 

Advanced 
CBT/WBT 
courseware  for 
distributed/distance 
learning 
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Data  Collection 

Archival  data  +  Coordination 
with  range  managers/directors 

Archival  data  +  Coordination 
with  range  and  equipment 
facilities  managers/directors 

Archival  data  4-  Classroom 

observation 

Applicable  To 

MOSQ  instruction  (resident 
and  non-resident) 

MOSQ  instruction  (simulation- 
assisted't 

r 

MOSQ  instruction  (simulation- 
assisted) 

Metric 

Ratio  of  proponent 
range  supplies  (e.g., 
ammunition,  fuel, 
etc.)  used  per 
exercise  before  and 

1  after  LLC 

implementation 
(assumes  that 
enhanced 

competency  enabled 
by  LLC  will  make 
performance  more 
efficient  during 
capstone  field 
exercises  for  courses 
taught  via  the  LLC) 

Ratio  of  proponent 
range  or  other 
equipment  (e.g., 
digital  systems) 
requirements  before 
and  after  LLC 
implementation 
(assumes  that  use  of 
simulated  equipment 
in  the  LLC  will 
reduce  the  need  for 
actual  equipment 
located  at  the 
schoolhouse) 

%  reduction  in  time 
spent  on  equipment 

Metric  Category 

Decreased  range 

equipment/supplies 

requirements 

